Note: This is test shard 4 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN ] LocationAwareRebalancingBasicTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 13:59:39.176326 8538 test_util.cc:268] Using random seed: 60430705
I20240502 13:59:39.183626 8538 ts_itest-base.cc:112] Starting cluster with:
I20240502 13:59:39.183737 8538 ts_itest-base.cc:113] --------------
I20240502 13:59:39.183780 8538 ts_itest-base.cc:114] 6 tablet servers
I20240502 13:59:39.183815 8538 ts_itest-base.cc:115] 3 replicas per TS
I20240502 13:59:39.183830 8538 ts_itest-base.cc:116] --------------
2024-05-02T13:59:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T13:59:39Z Disabled control of system clock
I20240502 13:59:39.192896 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.86.190:40597
--webserver_interface=127.8.86.190
--webserver_port=0
--builtin_ntp_servers=127.8.86.148:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.86.190:40597
--location_mapping_cmd=/tmp/dist-test-task4LMC7s/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 13:59:39.290398 8550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:39.290588 8550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:39.290627 8550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:39.293534 8550 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:39.293591 8550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 13:59:39.293613 8550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:39.293632 8550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 13:59:39.293654 8550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 13:59:39.297578 8550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-task4LMC7s/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.8.86.190:40597
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.86.190:40597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.8.86.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.8550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:39.298429 8550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:39.299448 8550 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:39.306911 8558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:39.307948 8550 server_base.cc:1034] running on GCE node
W20240502 13:59:39.308491 8555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.309244 8556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:39.309697 8550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:39.310545 8550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:39.311690 8550 hybrid_clock.cc:648] HybridClock initialized: now 1714658379311674 us; error 30 us; skew 500 ppm
I20240502 13:59:39.328258 8550 webserver.cc:460] Webserver started at http://127.8.86.190:38895/ using document root <none> and password file <none>
I20240502 13:59:39.328801 8550 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:39.328861 8550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:39.329027 8550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:39.330864 8550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "65d1b70050584073b6a58e6ef213ef9b"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-h6v0"
I20240502 13:59:39.331300 8550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "65d1b70050584073b6a58e6ef213ef9b"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-h6v0"
I20240502 13:59:39.334420 8550 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 13:59:39.336475 8564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.337486 8550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:39.337600 8550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
uuid: "65d1b70050584073b6a58e6ef213ef9b"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-h6v0"
I20240502 13:59:39.337675 8550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:39.366850 8550 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:39.367399 8550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:39.367552 8550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:39.373628 8550 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.190:40597
I20240502 13:59:39.373659 8616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.190:40597 every 8 connection(s)
I20240502 13:59:39.374536 8550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 13:59:39.377024 8617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:39.377866 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 8550
I20240502 13:59:39.378001 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 13:59:39.382920 8617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b: Bootstrap starting.
I20240502 13:59:39.385324 8617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:39.385982 8617 log.cc:824] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:39.387634 8617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b: No bootstrap required, opened a new log
I20240502 13:59:39.390563 8617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d1b70050584073b6a58e6ef213ef9b" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 40597 } }
I20240502 13:59:39.390750 8617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:39.390820 8617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d1b70050584073b6a58e6ef213ef9b, State: Initialized, Role: FOLLOWER
I20240502 13:59:39.391281 8617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d1b70050584073b6a58e6ef213ef9b" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 40597 } }
I20240502 13:59:39.391434 8617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:39.391501 8617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:39.391598 8617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:39.392302 8617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d1b70050584073b6a58e6ef213ef9b" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 40597 } }
I20240502 13:59:39.392588 8617 leader_election.cc:302] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65d1b70050584073b6a58e6ef213ef9b; no voters:
I20240502 13:59:39.392848 8617 leader_election.cc:288] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:39.393172 8623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:39.393752 8617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b [sys.catalog]: configured and running, proceeding with master startup.
I20240502 13:59:39.393787 8623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b [term 1 LEADER]: Becoming Leader. State: Replica: 65d1b70050584073b6a58e6ef213ef9b, State: Running, Role: LEADER
I20240502 13:59:39.394073 8623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d1b70050584073b6a58e6ef213ef9b" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 40597 } }
I20240502 13:59:39.395488 8624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65d1b70050584073b6a58e6ef213ef9b. Latest consensus state: current_term: 1 leader_uuid: "65d1b70050584073b6a58e6ef213ef9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d1b70050584073b6a58e6ef213ef9b" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 40597 } } }
I20240502 13:59:39.395596 8624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:39.395848 8622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65d1b70050584073b6a58e6ef213ef9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d1b70050584073b6a58e6ef213ef9b" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 40597 } } }
I20240502 13:59:39.395942 8622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b [sys.catalog]: This master's current role is: LEADER
W20240502 13:59:39.397964 8637 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 13:59:39.398022 8637 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 13:59:39.398123 8636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 13:59:39.398910 8636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 13:59:39.403164 8636 catalog_manager.cc:1347] Generated new cluster ID: aebaaca8d92446c396e48a748deac770
I20240502 13:59:39.403225 8636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 13:59:39.411756 8636 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 13:59:39.412508 8636 catalog_manager.cc:1504] Loading token signing keys...
I20240502 13:59:39.420773 8636 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 65d1b70050584073b6a58e6ef213ef9b: Generated new TSK 0
I20240502 13:59:39.421310 8636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 13:59:39.426236 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.129:0
--local_ip_for_outbound_sockets=127.8.86.129
--webserver_interface=127.8.86.129
--webserver_port=0
--tserver_master_addrs=127.8.86.190:40597
--builtin_ntp_servers=127.8.86.148:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 13:59:39.532969 8641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:39.533172 8641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:39.533211 8641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:39.536163 8641 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:39.536224 8641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:39.536301 8641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.129
I20240502 13:59:39.539780 8641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.8.86.129
--webserver_port=0
--tserver_master_addrs=127.8.86.190:40597
--never_fsync=true
--heap_profile_path=/tmp/kudu.8641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.129
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:39.540560 8641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:39.541661 8641 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:39.549750 8646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.550274 8649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.551060 8647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:39.551873 8641 server_base.cc:1034] running on GCE node
I20240502 13:59:39.552214 8641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:39.552704 8641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:39.553848 8641 hybrid_clock.cc:648] HybridClock initialized: now 1714658379553831 us; error 35 us; skew 500 ppm
I20240502 13:59:39.555563 8641 webserver.cc:460] Webserver started at http://127.8.86.129:42711/ using document root <none> and password file <none>
I20240502 13:59:39.556123 8641 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:39.556186 8641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:39.556356 8641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:39.557901 8641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4397244002fd4e0f9246b55727b536fe"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-h6v0"
I20240502 13:59:39.558408 8641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4397244002fd4e0f9246b55727b536fe"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-h6v0"
I20240502 13:59:39.561354 8641 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20240502 13:59:39.563486 8655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.564405 8641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:39.564529 8641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4397244002fd4e0f9246b55727b536fe"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-h6v0"
I20240502 13:59:39.564615 8641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:39.599784 8641 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:39.600415 8641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:39.600567 8641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:39.601089 8641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:39.601975 8641 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:39.602022 8641 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.602077 8641 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:39.602103 8641 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.609975 8641 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.129:41573
I20240502 13:59:39.610028 8768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.129:41573 every 8 connection(s)
I20240502 13:59:39.610841 8641 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 13:59:39.611892 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 8641
I20240502 13:59:39.611990 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 13:59:39.614215 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.130:0
--local_ip_for_outbound_sockets=127.8.86.130
--webserver_interface=127.8.86.130
--webserver_port=0
--tserver_master_addrs=127.8.86.190:40597
--builtin_ntp_servers=127.8.86.148:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 13:59:39.620262 8769 heartbeater.cc:347] Connected to a master server at 127.8.86.190:40597
I20240502 13:59:39.620477 8769 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:39.620955 8769 heartbeater.cc:511] Master 127.8.86.190:40597 requested a full tablet report, sending...
I20240502 13:59:39.664995 8581 ts_manager.cc:194] Registered new tserver with Master: 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573)
I20240502 13:59:39.666379 8581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.129:36111
W20240502 13:59:39.732618 8772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:39.732803 8772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:39.732848 8772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:39.735769 8772 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:39.735829 8772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:39.735934 8772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.130
I20240502 13:59:39.739437 8772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.8.86.130
--webserver_port=0
--tserver_master_addrs=127.8.86.190:40597
--never_fsync=true
--heap_profile_path=/tmp/kudu.8772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.130
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:39.740192 8772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:39.741299 8772 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:39.749195 8781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.749974 8778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.750627 8779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:39.751452 8772 server_base.cc:1034] running on GCE node
I20240502 13:59:39.751757 8772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:39.752246 8772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:39.753397 8772 hybrid_clock.cc:648] HybridClock initialized: now 1714658379753383 us; error 24 us; skew 500 ppm
I20240502 13:59:39.755218 8772 webserver.cc:460] Webserver started at http://127.8.86.130:35931/ using document root <none> and password file <none>
I20240502 13:59:39.755782 8772 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:39.755838 8772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:39.756012 8772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:39.757583 8772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "379b5e0287244d6b94265ccffa51fc05"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-h6v0"
I20240502 13:59:39.758025 8772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "379b5e0287244d6b94265ccffa51fc05"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-h6v0"
I20240502 13:59:39.761090 8772 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 13:59:39.763137 8787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.764056 8772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:39.764161 8772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "379b5e0287244d6b94265ccffa51fc05"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-h6v0"
I20240502 13:59:39.764236 8772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:39.796209 8772 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:39.796819 8772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:39.796986 8772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:39.797489 8772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:39.798354 8772 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:39.798398 8772 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.798449 8772 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:39.798472 8772 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.806550 8772 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.130:41939
I20240502 13:59:39.806576 8900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.130:41939 every 8 connection(s)
I20240502 13:59:39.807384 8772 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 13:59:39.809810 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 8772
I20240502 13:59:39.809923 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 13:59:39.812206 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.131:0
--local_ip_for_outbound_sockets=127.8.86.131
--webserver_interface=127.8.86.131
--webserver_port=0
--tserver_master_addrs=127.8.86.190:40597
--builtin_ntp_servers=127.8.86.148:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 13:59:39.816277 8901 heartbeater.cc:347] Connected to a master server at 127.8.86.190:40597
I20240502 13:59:39.816470 8901 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:39.816923 8901 heartbeater.cc:511] Master 127.8.86.190:40597 requested a full tablet report, sending...
I20240502 13:59:39.849737 8581 ts_manager.cc:194] Registered new tserver with Master: 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939)
I20240502 13:59:39.850504 8581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.130:39249
W20240502 13:59:39.913317 8904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:39.913534 8904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:39.913599 8904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:39.916522 8904 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:39.916592 8904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:39.916676 8904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.131
I20240502 13:59:39.920122 8904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.8.86.131
--webserver_port=0
--tserver_master_addrs=127.8.86.190:40597
--never_fsync=true
--heap_profile_path=/tmp/kudu.8904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.131
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:39.920899 8904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:39.921883 8904 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:39.929509 8911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.930193 8913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.930835 8910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:39.931687 8904 server_base.cc:1034] running on GCE node
I20240502 13:59:39.932004 8904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:39.932475 8904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:39.933610 8904 hybrid_clock.cc:648] HybridClock initialized: now 1714658379933600 us; error 23 us; skew 500 ppm
I20240502 13:59:39.935305 8904 webserver.cc:460] Webserver started at http://127.8.86.131:44599/ using document root <none> and password file <none>
I20240502 13:59:39.935819 8904 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:39.935873 8904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:39.936039 8904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:39.937537 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-h6v0"
I20240502 13:59:39.937978 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-h6v0"
I20240502 13:59:39.941031 8904 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240502 13:59:39.943027 8919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.943845 8904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:39.943933 8904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-h6v0"
I20240502 13:59:39.944007 8904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:39.973536 8904 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:39.974148 8904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:39.974303 8904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:39.974773 8904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:39.975517 8904 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:39.975558 8904 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.975602 8904 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:39.975631 8904 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.983716 8904 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.131:39873
I20240502 13:59:39.983753 9032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.131:39873 every 8 connection(s)
I20240502 13:59:39.984570 8904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 13:59:39.988343 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 8904
I20240502 13:59:39.988433 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 13:59:39.990423 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.132:0
--local_ip_for_outbound_sockets=127.8.86.132
--webserver_interface=127.8.86.132
--webserver_port=0
--tserver_master_addrs=127.8.86.190:40597
--builtin_ntp_servers=127.8.86.148:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 13:59:39.992945 9033 heartbeater.cc:347] Connected to a master server at 127.8.86.190:40597
I20240502 13:59:39.993129 9033 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:39.993568 9033 heartbeater.cc:511] Master 127.8.86.190:40597 requested a full tablet report, sending...
I20240502 13:59:40.040768 8581 ts_manager.cc:194] Registered new tserver with Master: 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873)
I20240502 13:59:40.041555 8581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.131:37469
W20240502 13:59:40.106781 9036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:40.106981 9036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:40.107064 9036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:40.109795 9036 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:40.109856 9036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:40.109938 9036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.132
I20240502 13:59:40.113305 9036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.86.132
--webserver_port=0
--tserver_master_addrs=127.8.86.190:40597
--never_fsync=true
--heap_profile_path=/tmp/kudu.9036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.132
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:40.114064 9036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:40.115104 9036 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:40.122709 9045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.123482 9043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.124106 9042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:40.124922 9036 server_base.cc:1034] running on GCE node
I20240502 13:59:40.125223 9036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:40.125730 9036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:40.126889 9036 hybrid_clock.cc:648] HybridClock initialized: now 1714658380126867 us; error 25 us; skew 500 ppm
I20240502 13:59:40.128465 9036 webserver.cc:460] Webserver started at http://127.8.86.132:37985/ using document root <none> and password file <none>
I20240502 13:59:40.128947 9036 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:40.129004 9036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:40.129173 9036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:40.130681 9036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2695a61796d043aa823fe3ee966a7541"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-h6v0"
I20240502 13:59:40.131107 9036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2695a61796d043aa823fe3ee966a7541"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-h6v0"
I20240502 13:59:40.134003 9036 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 13:59:40.135892 9051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.136780 9036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:40.136881 9036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2695a61796d043aa823fe3ee966a7541"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-h6v0"
I20240502 13:59:40.136963 9036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:40.152129 9036 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:40.152588 9036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:40.152714 9036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:40.153127 9036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:40.153841 9036 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:40.153882 9036 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.153928 9036 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:40.153954 9036 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.161506 9036 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.132:35215
I20240502 13:59:40.161551 9164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.132:35215 every 8 connection(s)
I20240502 13:59:40.162263 9036 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 13:59:40.165719 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 9036
I20240502 13:59:40.165834 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 13:59:40.168025 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.133:0
--local_ip_for_outbound_sockets=127.8.86.133
--webserver_interface=127.8.86.133
--webserver_port=0
--tserver_master_addrs=127.8.86.190:40597
--builtin_ntp_servers=127.8.86.148:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 13:59:40.170847 9165 heartbeater.cc:347] Connected to a master server at 127.8.86.190:40597
I20240502 13:59:40.171044 9165 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:40.171537 9165 heartbeater.cc:511] Master 127.8.86.190:40597 requested a full tablet report, sending...
I20240502 13:59:40.208817 8581 ts_manager.cc:194] Registered new tserver with Master: 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215)
I20240502 13:59:40.209553 8581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.132:49129
W20240502 13:59:40.275269 9168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:40.275449 9168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:40.275488 9168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:40.278409 9168 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:40.278460 9168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:40.278544 9168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.133
I20240502 13:59:40.281855 9168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.86.133
--webserver_port=0
--tserver_master_addrs=127.8.86.190:40597
--never_fsync=true
--heap_profile_path=/tmp/kudu.9168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.133
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:40.282682 9168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:40.283671 9168 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:40.291390 9174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.292106 9175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.292914 9177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:40.293665 9168 server_base.cc:1034] running on GCE node
I20240502 13:59:40.293998 9168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:40.294553 9168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:40.295701 9168 hybrid_clock.cc:648] HybridClock initialized: now 1714658380295689 us; error 43 us; skew 500 ppm
I20240502 13:59:40.297384 9168 webserver.cc:460] Webserver started at http://127.8.86.133:41759/ using document root <none> and password file <none>
I20240502 13:59:40.297850 9168 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:40.297902 9168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:40.298048 9168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:40.299484 9168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b0a54f7311df4dd2be068d3b15a4a360"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-h6v0"
I20240502 13:59:40.299919 9168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b0a54f7311df4dd2be068d3b15a4a360"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-h6v0"
I20240502 13:59:40.302780 9168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 13:59:40.304641 9183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.305585 9168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:40.305691 9168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b0a54f7311df4dd2be068d3b15a4a360"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-h6v0"
I20240502 13:59:40.305761 9168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:40.332291 9168 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:40.332793 9168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:40.332912 9168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:40.333356 9168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:40.334120 9168 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:40.334169 9168 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.334210 9168 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:40.334233 9168 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.342521 9168 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.133:33699
I20240502 13:59:40.342548 9296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.133:33699 every 8 connection(s)
I20240502 13:59:40.343362 9168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 13:59:40.343511 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 9168
I20240502 13:59:40.343600 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 13:59:40.345929 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.134:0
--local_ip_for_outbound_sockets=127.8.86.134
--webserver_interface=127.8.86.134
--webserver_port=0
--tserver_master_addrs=127.8.86.190:40597
--builtin_ntp_servers=127.8.86.148:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 13:59:40.352917 9297 heartbeater.cc:347] Connected to a master server at 127.8.86.190:40597
I20240502 13:59:40.353159 9297 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:40.353821 9297 heartbeater.cc:511] Master 127.8.86.190:40597 requested a full tablet report, sending...
I20240502 13:59:40.391466 8581 ts_manager.cc:194] Registered new tserver with Master: b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699)
I20240502 13:59:40.392393 8581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.133:40761
W20240502 13:59:40.455467 9300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:40.455651 9300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:40.455691 9300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:40.458607 9300 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:40.458657 9300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:40.458739 9300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.134
I20240502 13:59:40.462025 9300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.8.86.134
--webserver_port=0
--tserver_master_addrs=127.8.86.190:40597
--never_fsync=true
--heap_profile_path=/tmp/kudu.9300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.134
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:40.462816 9300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:40.464089 9300 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:40.472358 9306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.472856 9307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:40.473690 9309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:40.474488 9300 server_base.cc:1034] running on GCE node
I20240502 13:59:40.474789 9300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:40.475239 9300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:40.476361 9300 hybrid_clock.cc:648] HybridClock initialized: now 1714658380476345 us; error 32 us; skew 500 ppm
I20240502 13:59:40.478063 9300 webserver.cc:460] Webserver started at http://127.8.86.134:37093/ using document root <none> and password file <none>
I20240502 13:59:40.478587 9300 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:40.478658 9300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:40.478824 9300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:40.480293 9300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "833c2ad4636d4d8e9861339a4ad9e80e"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-h6v0"
I20240502 13:59:40.480701 9300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "833c2ad4636d4d8e9861339a4ad9e80e"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-h6v0"
I20240502 13:59:40.483951 9300 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 13:59:40.486073 9315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.487051 9300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:40.487147 9300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "833c2ad4636d4d8e9861339a4ad9e80e"
format_stamp: "Formatted at 2024-05-02 13:59:40 on dist-test-slave-h6v0"
I20240502 13:59:40.487232 9300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:40.496325 9300 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:40.496834 9300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:40.496979 9300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:40.497437 9300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:40.498203 9300 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:40.498245 9300 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.498302 9300 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:40.498337 9300 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:40.506376 9300 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.134:35071
I20240502 13:59:40.506446 9428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.134:35071 every 8 connection(s)
I20240502 13:59:40.507512 9300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 13:59:40.511857 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 9300
I20240502 13:59:40.511969 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 13:59:40.516988 9429 heartbeater.cc:347] Connected to a master server at 127.8.86.190:40597
I20240502 13:59:40.517192 9429 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:40.517613 9429 heartbeater.cc:511] Master 127.8.86.190:40597 requested a full tablet report, sending...
I20240502 13:59:40.550033 8581 ts_manager.cc:194] Registered new tserver with Master: 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134:35071)
I20240502 13:59:40.550524 8538 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 13:59:40.550776 8581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.134:49521
I20240502 13:59:40.561172 8538 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 13:59:40.669253 8769 heartbeater.cc:503] Master 127.8.86.190:40597 was elected leader, sending a full tablet report...
I20240502 13:59:40.852488 8901 heartbeater.cc:503] Master 127.8.86.190:40597 was elected leader, sending a full tablet report...
I20240502 13:59:41.043550 9033 heartbeater.cc:503] Master 127.8.86.190:40597 was elected leader, sending a full tablet report...
I20240502 13:59:41.211683 9165 heartbeater.cc:503] Master 127.8.86.190:40597 was elected leader, sending a full tablet report...
I20240502 13:59:41.395085 9297 heartbeater.cc:503] Master 127.8.86.190:40597 was elected leader, sending a full tablet report...
I20240502 13:59:41.552825 9429 heartbeater.cc:503] Master 127.8.86.190:40597 was elected leader, sending a full tablet report...
I20240502 13:59:44.313510 8581 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54006:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:44.335904 9363 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1f2054645e4f2bb2e26831f698e871 (DEFAULT_TABLE table=rebalance_test_table_0 [id=28b1255325114b8e9bce007d4d2c08d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.336913 9363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1f2054645e4f2bb2e26831f698e871. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.338199 8702 tablet_service.cc:1467] Processing CreateTablet for tablet 96f107cb7739454f906466f5c061d3fc (DEFAULT_TABLE table=rebalance_test_table_0 [id=28b1255325114b8e9bce007d4d2c08d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.339049 8702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96f107cb7739454f906466f5c061d3fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.338896 9099 tablet_service.cc:1467] Processing CreateTablet for tablet 72e38e431ec0424ba8b1e7813e26aa27 (DEFAULT_TABLE table=rebalance_test_table_0 [id=28b1255325114b8e9bce007d4d2c08d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.339634 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e38e431ec0424ba8b1e7813e26aa27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.339977 8835 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1f2054645e4f2bb2e26831f698e871 (DEFAULT_TABLE table=rebalance_test_table_0 [id=28b1255325114b8e9bce007d4d2c08d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.340634 8835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1f2054645e4f2bb2e26831f698e871. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.342554 8703 tablet_service.cc:1467] Processing CreateTablet for tablet 72e38e431ec0424ba8b1e7813e26aa27 (DEFAULT_TABLE table=rebalance_test_table_0 [id=28b1255325114b8e9bce007d4d2c08d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.344969 9231 tablet_service.cc:1467] Processing CreateTablet for tablet 72e38e431ec0424ba8b1e7813e26aa27 (DEFAULT_TABLE table=rebalance_test_table_0 [id=28b1255325114b8e9bce007d4d2c08d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.345747 9231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e38e431ec0424ba8b1e7813e26aa27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.346900 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e38e431ec0424ba8b1e7813e26aa27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.347222 9230 tablet_service.cc:1467] Processing CreateTablet for tablet 96f107cb7739454f906466f5c061d3fc (DEFAULT_TABLE table=rebalance_test_table_0 [id=28b1255325114b8e9bce007d4d2c08d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.347460 9230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96f107cb7739454f906466f5c061d3fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.348680 8967 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1f2054645e4f2bb2e26831f698e871 (DEFAULT_TABLE table=rebalance_test_table_0 [id=28b1255325114b8e9bce007d4d2c08d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.349426 8967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1f2054645e4f2bb2e26831f698e871. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.349375 8965 tablet_service.cc:1467] Processing CreateTablet for tablet 96f107cb7739454f906466f5c061d3fc (DEFAULT_TABLE table=rebalance_test_table_0 [id=28b1255325114b8e9bce007d4d2c08d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.350086 8965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96f107cb7739454f906466f5c061d3fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.353648 9455 tablet_bootstrap.cc:492] T 96f107cb7739454f906466f5c061d3fc P 4397244002fd4e0f9246b55727b536fe: Bootstrap starting.
I20240502 13:59:44.354598 9454 tablet_bootstrap.cc:492] T 5b1f2054645e4f2bb2e26831f698e871 P 379b5e0287244d6b94265ccffa51fc05: Bootstrap starting.
I20240502 13:59:44.354683 9456 tablet_bootstrap.cc:492] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e: Bootstrap starting.
I20240502 13:59:44.355592 9455 tablet_bootstrap.cc:654] T 96f107cb7739454f906466f5c061d3fc P 4397244002fd4e0f9246b55727b536fe: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.356410 9455 log.cc:824] T 96f107cb7739454f906466f5c061d3fc P 4397244002fd4e0f9246b55727b536fe: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:44.356665 9456 tablet_bootstrap.cc:654] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.356707 9459 tablet_bootstrap.cc:492] T 5b1f2054645e4f2bb2e26831f698e871 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Bootstrap starting.
I20240502 13:59:44.357095 9457 tablet_bootstrap.cc:492] T 72e38e431ec0424ba8b1e7813e26aa27 P 2695a61796d043aa823fe3ee966a7541: Bootstrap starting.
I20240502 13:59:44.357376 9454 tablet_bootstrap.cc:654] T 5b1f2054645e4f2bb2e26831f698e871 P 379b5e0287244d6b94265ccffa51fc05: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.357528 9456 log.cc:824] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:44.358213 9454 log.cc:824] T 5b1f2054645e4f2bb2e26831f698e871 P 379b5e0287244d6b94265ccffa51fc05: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:44.358820 9459 tablet_bootstrap.cc:654] T 5b1f2054645e4f2bb2e26831f698e871 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.358959 9457 tablet_bootstrap.cc:654] T 72e38e431ec0424ba8b1e7813e26aa27 P 2695a61796d043aa823fe3ee966a7541: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.359292 9456 tablet_bootstrap.cc:492] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e: No bootstrap required, opened a new log
I20240502 13:59:44.359475 9456 ts_tablet_manager.cc:1367] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 13:59:44.359535 9459 log.cc:824] T 5b1f2054645e4f2bb2e26831f698e871 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:44.359549 9457 log.cc:824] T 72e38e431ec0424ba8b1e7813e26aa27 P 2695a61796d043aa823fe3ee966a7541: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:44.360018 9458 tablet_bootstrap.cc:492] T 96f107cb7739454f906466f5c061d3fc P b0a54f7311df4dd2be068d3b15a4a360: Bootstrap starting.
I20240502 13:59:44.361325 9455 tablet_bootstrap.cc:492] T 96f107cb7739454f906466f5c061d3fc P 4397244002fd4e0f9246b55727b536fe: No bootstrap required, opened a new log
I20240502 13:59:44.361462 9457 tablet_bootstrap.cc:492] T 72e38e431ec0424ba8b1e7813e26aa27 P 2695a61796d043aa823fe3ee966a7541: No bootstrap required, opened a new log
I20240502 13:59:44.361531 9455 ts_tablet_manager.cc:1367] T 96f107cb7739454f906466f5c061d3fc P 4397244002fd4e0f9246b55727b536fe: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 13:59:44.361667 9458 tablet_bootstrap.cc:654] T 96f107cb7739454f906466f5c061d3fc P b0a54f7311df4dd2be068d3b15a4a360: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.361737 9457 ts_tablet_manager.cc:1367] T 72e38e431ec0424ba8b1e7813e26aa27 P 2695a61796d043aa823fe3ee966a7541: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 13:59:44.362681 9458 log.cc:824] T 96f107cb7739454f906466f5c061d3fc P b0a54f7311df4dd2be068d3b15a4a360: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:44.363396 9456 raft_consensus.cc:348] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:44.363700 9456 raft_consensus.cc:374] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.363811 9456 raft_consensus.cc:729] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.364326 9456 consensus_queue.cc:260] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:44.364639 9459 tablet_bootstrap.cc:492] T 5b1f2054645e4f2bb2e26831f698e871 P 66ef9d15c0bd46fbb23d5ba11a085ba6: No bootstrap required, opened a new log
I20240502 13:59:44.364662 9454 tablet_bootstrap.cc:492] T 5b1f2054645e4f2bb2e26831f698e871 P 379b5e0287244d6b94265ccffa51fc05: No bootstrap required, opened a new log
I20240502 13:59:44.364805 9459 ts_tablet_manager.cc:1367] T 5b1f2054645e4f2bb2e26831f698e871 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 13:59:44.364815 9454 ts_tablet_manager.cc:1367] T 5b1f2054645e4f2bb2e26831f698e871 P 379b5e0287244d6b94265ccffa51fc05: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 13:59:44.365471 9456 ts_tablet_manager.cc:1398] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20240502 13:59:44.366444 9455 raft_consensus.cc:348] T 96f107cb7739454f906466f5c061d3fc P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:44.366703 9455 raft_consensus.cc:374] T 96f107cb7739454f906466f5c061d3fc P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.366808 9455 raft_consensus.cc:729] T 96f107cb7739454f906466f5c061d3fc P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.367381 9455 consensus_queue.cc:260] T 96f107cb7739454f906466f5c061d3fc P 4397244002fd4e0f9246b55727b536fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:44.368460 9454 raft_consensus.cc:348] T 5b1f2054645e4f2bb2e26831f698e871 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:44.368613 9455 ts_tablet_manager.cc:1398] T 96f107cb7739454f906466f5c061d3fc P 4397244002fd4e0f9246b55727b536fe: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 13:59:44.368683 9454 raft_consensus.cc:374] T 5b1f2054645e4f2bb2e26831f698e871 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.368604 9459 raft_consensus.cc:348] T 5b1f2054645e4f2bb2e26831f698e871 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:44.368774 9454 raft_consensus.cc:729] T 5b1f2054645e4f2bb2e26831f698e871 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.368906 9459 raft_consensus.cc:374] T 5b1f2054645e4f2bb2e26831f698e871 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.369009 9459 raft_consensus.cc:729] T 5b1f2054645e4f2bb2e26831f698e871 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.369094 9455 tablet_bootstrap.cc:492] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe: Bootstrap starting.
I20240502 13:59:44.369164 9454 consensus_queue.cc:260] T 5b1f2054645e4f2bb2e26831f698e871 P 379b5e0287244d6b94265ccffa51fc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:44.369457 9459 consensus_queue.cc:260] T 5b1f2054645e4f2bb2e26831f698e871 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:44.369952 9454 ts_tablet_manager.cc:1398] T 5b1f2054645e4f2bb2e26831f698e871 P 379b5e0287244d6b94265ccffa51fc05: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 13:59:44.370148 9455 tablet_bootstrap.cc:654] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.370244 9459 ts_tablet_manager.cc:1398] T 5b1f2054645e4f2bb2e26831f698e871 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 13:59:44.370606 9459 tablet_bootstrap.cc:492] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6: Bootstrap starting.
I20240502 13:59:44.371511 9459 tablet_bootstrap.cc:654] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.372751 9459 tablet_bootstrap.cc:492] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6: No bootstrap required, opened a new log
I20240502 13:59:44.372838 9459 ts_tablet_manager.cc:1367] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:44.373361 9459 raft_consensus.cc:348] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:44.373441 9459 raft_consensus.cc:374] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.373482 9459 raft_consensus.cc:729] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.373607 9459 consensus_queue.cc:260] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:44.373824 9459 ts_tablet_manager.cc:1398] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:44.373745 9457 raft_consensus.cc:348] T 72e38e431ec0424ba8b1e7813e26aa27 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:44.373936 9457 raft_consensus.cc:374] T 72e38e431ec0424ba8b1e7813e26aa27 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.374008 9457 raft_consensus.cc:729] T 72e38e431ec0424ba8b1e7813e26aa27 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.374357 9458 tablet_bootstrap.cc:492] T 96f107cb7739454f906466f5c061d3fc P b0a54f7311df4dd2be068d3b15a4a360: No bootstrap required, opened a new log
I20240502 13:59:44.374392 9457 consensus_queue.cc:260] T 72e38e431ec0424ba8b1e7813e26aa27 P 2695a61796d043aa823fe3ee966a7541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:44.374550 9458 ts_tablet_manager.cc:1367] T 96f107cb7739454f906466f5c061d3fc P b0a54f7311df4dd2be068d3b15a4a360: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240502 13:59:44.374686 9455 tablet_bootstrap.cc:492] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe: No bootstrap required, opened a new log
I20240502 13:59:44.374776 9455 ts_tablet_manager.cc:1367] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240502 13:59:44.375097 9457 ts_tablet_manager.cc:1398] T 72e38e431ec0424ba8b1e7813e26aa27 P 2695a61796d043aa823fe3ee966a7541: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240502 13:59:44.375239 9455 raft_consensus.cc:348] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:44.376060 9455 raft_consensus.cc:374] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.376132 9455 raft_consensus.cc:729] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.376261 9455 consensus_queue.cc:260] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:44.376686 9455 ts_tablet_manager.cc:1398] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:44.379591 9458 raft_consensus.cc:348] T 96f107cb7739454f906466f5c061d3fc P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:44.379793 9458 raft_consensus.cc:374] T 96f107cb7739454f906466f5c061d3fc P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.379853 9458 raft_consensus.cc:729] T 96f107cb7739454f906466f5c061d3fc P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.380309 9458 consensus_queue.cc:260] T 96f107cb7739454f906466f5c061d3fc P b0a54f7311df4dd2be068d3b15a4a360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:44.381078 9458 ts_tablet_manager.cc:1398] T 96f107cb7739454f906466f5c061d3fc P b0a54f7311df4dd2be068d3b15a4a360: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 13:59:44.381408 9458 tablet_bootstrap.cc:492] T 72e38e431ec0424ba8b1e7813e26aa27 P b0a54f7311df4dd2be068d3b15a4a360: Bootstrap starting.
I20240502 13:59:44.382189 9458 tablet_bootstrap.cc:654] T 72e38e431ec0424ba8b1e7813e26aa27 P b0a54f7311df4dd2be068d3b15a4a360: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.383257 9458 tablet_bootstrap.cc:492] T 72e38e431ec0424ba8b1e7813e26aa27 P b0a54f7311df4dd2be068d3b15a4a360: No bootstrap required, opened a new log
I20240502 13:59:44.383344 9458 ts_tablet_manager.cc:1367] T 72e38e431ec0424ba8b1e7813e26aa27 P b0a54f7311df4dd2be068d3b15a4a360: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:44.383837 9458 raft_consensus.cc:348] T 72e38e431ec0424ba8b1e7813e26aa27 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:44.383934 9458 raft_consensus.cc:374] T 72e38e431ec0424ba8b1e7813e26aa27 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.383972 9458 raft_consensus.cc:729] T 72e38e431ec0424ba8b1e7813e26aa27 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.384095 9458 consensus_queue.cc:260] T 72e38e431ec0424ba8b1e7813e26aa27 P b0a54f7311df4dd2be068d3b15a4a360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:44.384294 9458 ts_tablet_manager.cc:1398] T 72e38e431ec0424ba8b1e7813e26aa27 P b0a54f7311df4dd2be068d3b15a4a360: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:44.411365 9466 raft_consensus.cc:482] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:44.411537 9466 raft_consensus.cc:504] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:44.412492 9466 leader_election.cc:288] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873), 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939)
W20240502 13:59:44.416000 9166 tablet.cc:2369] T 72e38e431ec0424ba8b1e7813e26aa27 P 2695a61796d043aa823fe3ee966a7541: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 13:59:44.416242 8855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1f2054645e4f2bb2e26831f698e871" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05" is_pre_election: true
I20240502 13:59:44.416524 8855 raft_consensus.cc:2454] T 5b1f2054645e4f2bb2e26831f698e871 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 0.
I20240502 13:59:44.416514 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1f2054645e4f2bb2e26831f698e871" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" is_pre_election: true
I20240502 13:59:44.416759 8987 raft_consensus.cc:2454] T 5b1f2054645e4f2bb2e26831f698e871 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 0.
I20240502 13:59:44.416986 9317 leader_election.cc:302] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 379b5e0287244d6b94265ccffa51fc05, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:44.417209 9466 raft_consensus.cc:2781] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:44.417279 9466 raft_consensus.cc:482] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:44.417320 9466 raft_consensus.cc:3037] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.418102 9466 raft_consensus.cc:504] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:44.418407 9466 leader_election.cc:288] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 election: Requested vote from peers 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873), 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939)
I20240502 13:59:44.418713 8855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1f2054645e4f2bb2e26831f698e871" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05"
I20240502 13:59:44.418704 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1f2054645e4f2bb2e26831f698e871" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6"
I20240502 13:59:44.418823 8987 raft_consensus.cc:3037] T 5b1f2054645e4f2bb2e26831f698e871 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.418823 8855 raft_consensus.cc:3037] T 5b1f2054645e4f2bb2e26831f698e871 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.419821 8987 raft_consensus.cc:2454] T 5b1f2054645e4f2bb2e26831f698e871 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 1.
I20240502 13:59:44.419821 8855 raft_consensus.cc:2454] T 5b1f2054645e4f2bb2e26831f698e871 P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 1.
I20240502 13:59:44.420086 9317 leader_election.cc:302] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 379b5e0287244d6b94265ccffa51fc05, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:44.420239 9466 raft_consensus.cc:2781] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:44.420418 9466 raft_consensus.cc:686] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 LEADER]: Becoming Leader. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Running, Role: LEADER
I20240502 13:59:44.420691 9466 consensus_queue.cc:237] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:44.423285 8578 catalog_manager.cc:5484] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134). New cstate: current_term: 1 leader_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:44.450577 9469 raft_consensus.cc:482] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:44.450685 9469 raft_consensus.cc:504] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:44.451555 9469 leader_election.cc:288] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573), b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699)
I20240502 13:59:44.455050 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f107cb7739454f906466f5c061d3fc" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe" is_pre_election: true
I20240502 13:59:44.455180 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f107cb7739454f906466f5c061d3fc" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360" is_pre_election: true
I20240502 13:59:44.455360 8723 raft_consensus.cc:2454] T 96f107cb7739454f906466f5c061d3fc P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 0.
I20240502 13:59:44.455423 9251 raft_consensus.cc:2454] T 96f107cb7739454f906466f5c061d3fc P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 0.
I20240502 13:59:44.455776 8921 leader_election.cc:302] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, 66ef9d15c0bd46fbb23d5ba11a085ba6; no voters:
I20240502 13:59:44.456017 9469 raft_consensus.cc:2781] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:44.456099 9469 raft_consensus.cc:482] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:44.456138 9469 raft_consensus.cc:3037] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.457201 9469 raft_consensus.cc:504] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:44.457542 9469 leader_election.cc:288] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 election: Requested vote from peers 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573), b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699)
I20240502 13:59:44.457904 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f107cb7739454f906466f5c061d3fc" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe"
I20240502 13:59:44.457903 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f107cb7739454f906466f5c061d3fc" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360"
I20240502 13:59:44.458025 9251 raft_consensus.cc:3037] T 96f107cb7739454f906466f5c061d3fc P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.458026 8723 raft_consensus.cc:3037] T 96f107cb7739454f906466f5c061d3fc P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.459125 9251 raft_consensus.cc:2454] T 96f107cb7739454f906466f5c061d3fc P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 1.
I20240502 13:59:44.459125 8723 raft_consensus.cc:2454] T 96f107cb7739454f906466f5c061d3fc P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 1.
I20240502 13:59:44.459421 8922 leader_election.cc:302] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66ef9d15c0bd46fbb23d5ba11a085ba6, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:44.459609 9469 raft_consensus.cc:2781] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:44.459820 9469 raft_consensus.cc:686] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 LEADER]: Becoming Leader. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Running, Role: LEADER
I20240502 13:59:44.460147 9469 consensus_queue.cc:237] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:44.463057 8578 catalog_manager.cc:5484] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131). New cstate: current_term: 1 leader_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:44.488512 9034 tablet.cc:2369] T 5b1f2054645e4f2bb2e26831f698e871 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 13:59:44.514910 9430 tablet.cc:2369] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 13:59:44.561295 8902 tablet.cc:2369] T 5b1f2054645e4f2bb2e26831f698e871 P 379b5e0287244d6b94265ccffa51fc05: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 13:59:44.596997 9298 tablet.cc:2369] T 72e38e431ec0424ba8b1e7813e26aa27 P b0a54f7311df4dd2be068d3b15a4a360: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 13:59:44.615176 8770 tablet.cc:2369] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 13:59:44.709304 9467 raft_consensus.cc:482] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:44.709443 9467 raft_consensus.cc:504] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:44.710567 9467 leader_election.cc:288] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699)
I20240502 13:59:44.714352 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e38e431ec0424ba8b1e7813e26aa27" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541" is_pre_election: true
I20240502 13:59:44.714392 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e38e431ec0424ba8b1e7813e26aa27" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360" is_pre_election: true
I20240502 13:59:44.714516 9251 raft_consensus.cc:2454] T 72e38e431ec0424ba8b1e7813e26aa27 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4397244002fd4e0f9246b55727b536fe in term 0.
I20240502 13:59:44.714634 9119 raft_consensus.cc:2454] T 72e38e431ec0424ba8b1e7813e26aa27 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4397244002fd4e0f9246b55727b536fe in term 0.
I20240502 13:59:44.714825 8658 leader_election.cc:302] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:44.715055 9467 raft_consensus.cc:2781] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:44.715116 9467 raft_consensus.cc:482] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:44.715143 9467 raft_consensus.cc:3037] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.715947 9467 raft_consensus.cc:504] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:44.716296 9467 leader_election.cc:288] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 election: Requested vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699)
I20240502 13:59:44.716558 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e38e431ec0424ba8b1e7813e26aa27" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541"
I20240502 13:59:44.716627 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e38e431ec0424ba8b1e7813e26aa27" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360"
I20240502 13:59:44.716661 9119 raft_consensus.cc:3037] T 72e38e431ec0424ba8b1e7813e26aa27 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.716728 9251 raft_consensus.cc:3037] T 72e38e431ec0424ba8b1e7813e26aa27 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.717394 9251 raft_consensus.cc:2454] T 72e38e431ec0424ba8b1e7813e26aa27 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4397244002fd4e0f9246b55727b536fe in term 1.
I20240502 13:59:44.717517 9119 raft_consensus.cc:2454] T 72e38e431ec0424ba8b1e7813e26aa27 P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4397244002fd4e0f9246b55727b536fe in term 1.
I20240502 13:59:44.717702 8658 leader_election.cc:302] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:44.717916 9467 raft_consensus.cc:2781] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:44.718012 9467 raft_consensus.cc:686] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe [term 1 LEADER]: Becoming Leader. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Running, Role: LEADER
I20240502 13:59:44.718276 9467 consensus_queue.cc:237] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:44.721318 8578 catalog_manager.cc:5484] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4397244002fd4e0f9246b55727b536fe (127.8.86.129). New cstate: current_term: 1 leader_uuid: "4397244002fd4e0f9246b55727b536fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:44.858422 9466 consensus_queue.cc:1035] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:44.864318 9478 consensus_queue.cc:1035] T 5b1f2054645e4f2bb2e26831f698e871 P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 13:59:44.882483 9487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:44.882699 9487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 13:59:44.884315 9469 consensus_queue.cc:1035] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 13:59:44.886374 9487 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 13:59:44.889114 9469 consensus_queue.cc:1035] T 96f107cb7739454f906466f5c061d3fc P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240502 13:59:44.987627 9251 raft_consensus.cc:1261] T 72e38e431ec0424ba8b1e7813e26aa27 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Refusing update from remote peer 4397244002fd4e0f9246b55727b536fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.988148 9467 consensus_queue.cc:1035] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:44.997256 9119 raft_consensus.cc:1261] T 72e38e431ec0424ba8b1e7813e26aa27 P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Refusing update from remote peer 4397244002fd4e0f9246b55727b536fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.997824 9467 consensus_queue.cc:1035] T 72e38e431ec0424ba8b1e7813e26aa27 P 4397244002fd4e0f9246b55727b536fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:45.023887 9501 mvcc.cc:205] Tried to move back new op lower bound from 7023240744901959680 to 7023240743808413696. Current Snapshot: MvccSnapshot[applied={T|T < 7023240744901959680}]
I20240502 13:59:45.024276 9499 mvcc.cc:205] Tried to move back new op lower bound from 7023240744901959680 to 7023240743808413696. Current Snapshot: MvccSnapshot[applied={T|T < 7023240744901959680}]
I20240502 13:59:45.048082 9538 mvcc.cc:205] Tried to move back new op lower bound from 7023240744901959680 to 7023240743808413696. Current Snapshot: MvccSnapshot[applied={T|T < 7023240744901959680}]
INSERT report
rows total: 2000
time total: 132.057 ms
time per row: 0.0660286 ms
I20240502 13:59:45.066241 8581 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54006:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:45.071799 9363 tablet_service.cc:1467] Processing CreateTablet for tablet fcf2b394ee794547824377bc3a4eae9a (DEFAULT_TABLE table=rebalance_test_table_1 [id=fd4dcd57c73b4a2aa21da79c231048ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.071933 9099 tablet_service.cc:1467] Processing CreateTablet for tablet fcf2b394ee794547824377bc3a4eae9a (DEFAULT_TABLE table=rebalance_test_table_1 [id=fd4dcd57c73b4a2aa21da79c231048ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.072166 9363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcf2b394ee794547824377bc3a4eae9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.072206 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcf2b394ee794547824377bc3a4eae9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.072876 8703 tablet_service.cc:1467] Processing CreateTablet for tablet 3a034d01e1b5402681c95d135646b88f (DEFAULT_TABLE table=rebalance_test_table_1 [id=fd4dcd57c73b4a2aa21da79c231048ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.073194 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a034d01e1b5402681c95d135646b88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.073596 9231 tablet_service.cc:1467] Processing CreateTablet for tablet 3a034d01e1b5402681c95d135646b88f (DEFAULT_TABLE table=rebalance_test_table_1 [id=fd4dcd57c73b4a2aa21da79c231048ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.073848 9231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a034d01e1b5402681c95d135646b88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.074539 9098 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1a21869b924bc79a6f82d5d0952dd7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fd4dcd57c73b4a2aa21da79c231048ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.074806 9098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1a21869b924bc79a6f82d5d0952dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.074819 9556 tablet_bootstrap.cc:492] T fcf2b394ee794547824377bc3a4eae9a P 2695a61796d043aa823fe3ee966a7541: Bootstrap starting.
I20240502 13:59:45.075840 9556 tablet_bootstrap.cc:654] T fcf2b394ee794547824377bc3a4eae9a P 2695a61796d043aa823fe3ee966a7541: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.076339 8835 tablet_service.cc:1467] Processing CreateTablet for tablet fcf2b394ee794547824377bc3a4eae9a (DEFAULT_TABLE table=rebalance_test_table_1 [id=fd4dcd57c73b4a2aa21da79c231048ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.076632 8835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcf2b394ee794547824377bc3a4eae9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.076709 9558 tablet_bootstrap.cc:492] T 3a034d01e1b5402681c95d135646b88f P 4397244002fd4e0f9246b55727b536fe: Bootstrap starting.
I20240502 13:59:45.077024 8965 tablet_service.cc:1467] Processing CreateTablet for tablet 3a034d01e1b5402681c95d135646b88f (DEFAULT_TABLE table=rebalance_test_table_1 [id=fd4dcd57c73b4a2aa21da79c231048ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.077301 8965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a034d01e1b5402681c95d135646b88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.077719 9558 tablet_bootstrap.cc:654] T 3a034d01e1b5402681c95d135646b88f P 4397244002fd4e0f9246b55727b536fe: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.078622 8834 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1a21869b924bc79a6f82d5d0952dd7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fd4dcd57c73b4a2aa21da79c231048ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.078877 8834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1a21869b924bc79a6f82d5d0952dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.079188 9562 tablet_bootstrap.cc:492] T fcf2b394ee794547824377bc3a4eae9a P 379b5e0287244d6b94265ccffa51fc05: Bootstrap starting.
I20240502 13:59:45.079835 9557 tablet_bootstrap.cc:492] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e: Bootstrap starting.
I20240502 13:59:45.079852 9558 tablet_bootstrap.cc:492] T 3a034d01e1b5402681c95d135646b88f P 4397244002fd4e0f9246b55727b536fe: No bootstrap required, opened a new log
I20240502 13:59:45.080005 9558 ts_tablet_manager.cc:1367] T 3a034d01e1b5402681c95d135646b88f P 4397244002fd4e0f9246b55727b536fe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 13:59:45.080052 9562 tablet_bootstrap.cc:654] T fcf2b394ee794547824377bc3a4eae9a P 379b5e0287244d6b94265ccffa51fc05: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.080564 9558 raft_consensus.cc:348] T 3a034d01e1b5402681c95d135646b88f P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:45.080691 9557 tablet_bootstrap.cc:654] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.080713 9558 raft_consensus.cc:374] T 3a034d01e1b5402681c95d135646b88f P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.080773 9558 raft_consensus.cc:729] T 3a034d01e1b5402681c95d135646b88f P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.080938 9558 consensus_queue.cc:260] T 3a034d01e1b5402681c95d135646b88f P 4397244002fd4e0f9246b55727b536fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:45.081221 9558 ts_tablet_manager.cc:1398] T 3a034d01e1b5402681c95d135646b88f P 4397244002fd4e0f9246b55727b536fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:45.081406 9363 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1a21869b924bc79a6f82d5d0952dd7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=fd4dcd57c73b4a2aa21da79c231048ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.081569 9559 tablet_bootstrap.cc:492] T 3a034d01e1b5402681c95d135646b88f P b0a54f7311df4dd2be068d3b15a4a360: Bootstrap starting.
I20240502 13:59:45.081614 9363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1a21869b924bc79a6f82d5d0952dd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.082361 9559 tablet_bootstrap.cc:654] T 3a034d01e1b5402681c95d135646b88f P b0a54f7311df4dd2be068d3b15a4a360: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.083436 9556 tablet_bootstrap.cc:492] T fcf2b394ee794547824377bc3a4eae9a P 2695a61796d043aa823fe3ee966a7541: No bootstrap required, opened a new log
I20240502 13:59:45.083499 9559 tablet_bootstrap.cc:492] T 3a034d01e1b5402681c95d135646b88f P b0a54f7311df4dd2be068d3b15a4a360: No bootstrap required, opened a new log
I20240502 13:59:45.083527 9556 ts_tablet_manager.cc:1367] T fcf2b394ee794547824377bc3a4eae9a P 2695a61796d043aa823fe3ee966a7541: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 13:59:45.083591 9559 ts_tablet_manager.cc:1367] T 3a034d01e1b5402681c95d135646b88f P b0a54f7311df4dd2be068d3b15a4a360: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:45.084085 9563 tablet_bootstrap.cc:492] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6: Bootstrap starting.
I20240502 13:59:45.084138 9556 raft_consensus.cc:348] T fcf2b394ee794547824377bc3a4eae9a P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.084252 9556 raft_consensus.cc:374] T fcf2b394ee794547824377bc3a4eae9a P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.084035 9559 raft_consensus.cc:348] T 3a034d01e1b5402681c95d135646b88f P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:45.084285 9556 raft_consensus.cc:729] T fcf2b394ee794547824377bc3a4eae9a P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.084302 9559 raft_consensus.cc:374] T 3a034d01e1b5402681c95d135646b88f P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.084343 9559 raft_consensus.cc:729] T 3a034d01e1b5402681c95d135646b88f P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.084404 9556 consensus_queue.cc:260] T fcf2b394ee794547824377bc3a4eae9a P 2695a61796d043aa823fe3ee966a7541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.084492 9559 consensus_queue.cc:260] T 3a034d01e1b5402681c95d135646b88f P b0a54f7311df4dd2be068d3b15a4a360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:45.084674 9556 ts_tablet_manager.cc:1398] T fcf2b394ee794547824377bc3a4eae9a P 2695a61796d043aa823fe3ee966a7541: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:45.084875 9556 tablet_bootstrap.cc:492] T 4f1a21869b924bc79a6f82d5d0952dd7 P 2695a61796d043aa823fe3ee966a7541: Bootstrap starting.
I20240502 13:59:45.085124 9563 tablet_bootstrap.cc:654] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.085733 9556 tablet_bootstrap.cc:654] T 4f1a21869b924bc79a6f82d5d0952dd7 P 2695a61796d043aa823fe3ee966a7541: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.086831 9557 tablet_bootstrap.cc:492] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e: No bootstrap required, opened a new log
I20240502 13:59:45.086921 9557 ts_tablet_manager.cc:1367] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 13:59:45.086673 9559 ts_tablet_manager.cc:1398] T 3a034d01e1b5402681c95d135646b88f P b0a54f7311df4dd2be068d3b15a4a360: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 13:59:45.087531 9557 raft_consensus.cc:348] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.087656 9557 raft_consensus.cc:374] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.087690 9557 raft_consensus.cc:729] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.087828 9557 consensus_queue.cc:260] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.088349 9563 tablet_bootstrap.cc:492] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6: No bootstrap required, opened a new log
I20240502 13:59:45.088474 9563 ts_tablet_manager.cc:1367] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 13:59:45.088680 9556 tablet_bootstrap.cc:492] T 4f1a21869b924bc79a6f82d5d0952dd7 P 2695a61796d043aa823fe3ee966a7541: No bootstrap required, opened a new log
I20240502 13:59:45.088763 9556 ts_tablet_manager.cc:1367] T 4f1a21869b924bc79a6f82d5d0952dd7 P 2695a61796d043aa823fe3ee966a7541: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 13:59:45.088930 9563 raft_consensus.cc:348] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:45.089025 9563 raft_consensus.cc:374] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.089046 9563 raft_consensus.cc:729] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.089177 9563 consensus_queue.cc:260] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:45.089253 9556 raft_consensus.cc:348] T 4f1a21869b924bc79a6f82d5d0952dd7 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.089352 9556 raft_consensus.cc:374] T 4f1a21869b924bc79a6f82d5d0952dd7 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.089381 9556 raft_consensus.cc:729] T 4f1a21869b924bc79a6f82d5d0952dd7 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.089418 9563 ts_tablet_manager.cc:1398] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:45.089519 9556 consensus_queue.cc:260] T 4f1a21869b924bc79a6f82d5d0952dd7 P 2695a61796d043aa823fe3ee966a7541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.089672 9557 ts_tablet_manager.cc:1398] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 13:59:45.089843 9557 tablet_bootstrap.cc:492] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e: Bootstrap starting.
I20240502 13:59:45.090682 9557 tablet_bootstrap.cc:654] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.091709 9557 tablet_bootstrap.cc:492] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e: No bootstrap required, opened a new log
I20240502 13:59:45.091806 9557 ts_tablet_manager.cc:1367] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:45.092200 9556 ts_tablet_manager.cc:1398] T 4f1a21869b924bc79a6f82d5d0952dd7 P 2695a61796d043aa823fe3ee966a7541: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 13:59:45.092252 9557 raft_consensus.cc:348] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.092326 9557 raft_consensus.cc:374] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.092346 9557 raft_consensus.cc:729] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.092444 9557 consensus_queue.cc:260] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.092656 9557 ts_tablet_manager.cc:1398] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:45.092804 9562 tablet_bootstrap.cc:492] T fcf2b394ee794547824377bc3a4eae9a P 379b5e0287244d6b94265ccffa51fc05: No bootstrap required, opened a new log
I20240502 13:59:45.092904 9562 ts_tablet_manager.cc:1367] T fcf2b394ee794547824377bc3a4eae9a P 379b5e0287244d6b94265ccffa51fc05: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20240502 13:59:45.093503 9562 raft_consensus.cc:348] T fcf2b394ee794547824377bc3a4eae9a P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.093636 9562 raft_consensus.cc:374] T fcf2b394ee794547824377bc3a4eae9a P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.093685 9562 raft_consensus.cc:729] T fcf2b394ee794547824377bc3a4eae9a P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.093829 9562 consensus_queue.cc:260] T fcf2b394ee794547824377bc3a4eae9a P 379b5e0287244d6b94265ccffa51fc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.094120 9562 ts_tablet_manager.cc:1398] T fcf2b394ee794547824377bc3a4eae9a P 379b5e0287244d6b94265ccffa51fc05: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:45.094353 9562 tablet_bootstrap.cc:492] T 4f1a21869b924bc79a6f82d5d0952dd7 P 379b5e0287244d6b94265ccffa51fc05: Bootstrap starting.
I20240502 13:59:45.095327 9562 tablet_bootstrap.cc:654] T 4f1a21869b924bc79a6f82d5d0952dd7 P 379b5e0287244d6b94265ccffa51fc05: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.096364 9562 tablet_bootstrap.cc:492] T 4f1a21869b924bc79a6f82d5d0952dd7 P 379b5e0287244d6b94265ccffa51fc05: No bootstrap required, opened a new log
I20240502 13:59:45.096439 9562 ts_tablet_manager.cc:1367] T 4f1a21869b924bc79a6f82d5d0952dd7 P 379b5e0287244d6b94265ccffa51fc05: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:45.096776 9562 raft_consensus.cc:348] T 4f1a21869b924bc79a6f82d5d0952dd7 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.096854 9562 raft_consensus.cc:374] T 4f1a21869b924bc79a6f82d5d0952dd7 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.096887 9562 raft_consensus.cc:729] T 4f1a21869b924bc79a6f82d5d0952dd7 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.097013 9562 consensus_queue.cc:260] T 4f1a21869b924bc79a6f82d5d0952dd7 P 379b5e0287244d6b94265ccffa51fc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.097242 9562 ts_tablet_manager.cc:1398] T 4f1a21869b924bc79a6f82d5d0952dd7 P 379b5e0287244d6b94265ccffa51fc05: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:45.200735 9482 raft_consensus.cc:482] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:45.200872 9482 raft_consensus.cc:504] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:45.201304 9482 leader_election.cc:288] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699), 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573)
I20240502 13:59:45.201695 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a034d01e1b5402681c95d135646b88f" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360" is_pre_election: true
I20240502 13:59:45.201752 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a034d01e1b5402681c95d135646b88f" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe" is_pre_election: true
I20240502 13:59:45.201823 9251 raft_consensus.cc:2454] T 3a034d01e1b5402681c95d135646b88f P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 0.
I20240502 13:59:45.201925 8723 raft_consensus.cc:2454] T 3a034d01e1b5402681c95d135646b88f P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 0.
I20240502 13:59:45.202118 8922 leader_election.cc:302] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66ef9d15c0bd46fbb23d5ba11a085ba6, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:45.202275 9482 raft_consensus.cc:2781] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:45.202365 9482 raft_consensus.cc:482] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:45.202410 9482 raft_consensus.cc:3037] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.203249 9482 raft_consensus.cc:504] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:45.203555 9482 leader_election.cc:288] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 election: Requested vote from peers b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699), 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573)
I20240502 13:59:45.203862 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a034d01e1b5402681c95d135646b88f" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360"
I20240502 13:59:45.203971 9251 raft_consensus.cc:3037] T 3a034d01e1b5402681c95d135646b88f P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.203951 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a034d01e1b5402681c95d135646b88f" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe"
I20240502 13:59:45.204074 8723 raft_consensus.cc:3037] T 3a034d01e1b5402681c95d135646b88f P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.204738 9251 raft_consensus.cc:2454] T 3a034d01e1b5402681c95d135646b88f P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 1.
I20240502 13:59:45.204828 8723 raft_consensus.cc:2454] T 3a034d01e1b5402681c95d135646b88f P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 1.
I20240502 13:59:45.205029 8922 leader_election.cc:302] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66ef9d15c0bd46fbb23d5ba11a085ba6, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:45.205171 9482 raft_consensus.cc:2781] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:45.205302 9482 raft_consensus.cc:686] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 LEADER]: Becoming Leader. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Running, Role: LEADER
I20240502 13:59:45.205456 9482 consensus_queue.cc:237] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:45.207082 8578 catalog_manager.cc:5484] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131). New cstate: current_term: 1 leader_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:45.229965 9547 raft_consensus.cc:482] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:45.230104 9547 raft_consensus.cc:504] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.230599 9547 leader_election.cc:288] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939)
I20240502 13:59:45.231434 8855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1a21869b924bc79a6f82d5d0952dd7" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05" is_pre_election: true
I20240502 13:59:45.231627 8855 raft_consensus.cc:2454] T 4f1a21869b924bc79a6f82d5d0952dd7 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 0.
I20240502 13:59:45.231968 9317 leader_election.cc:302] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 379b5e0287244d6b94265ccffa51fc05, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:45.232133 9547 raft_consensus.cc:2781] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:45.232208 9547 raft_consensus.cc:482] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:45.232244 9547 raft_consensus.cc:3037] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.233152 9547 raft_consensus.cc:504] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.233516 9547 leader_election.cc:288] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 election: Requested vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939)
I20240502 13:59:45.233877 8855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1a21869b924bc79a6f82d5d0952dd7" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05"
I20240502 13:59:45.233983 8855 raft_consensus.cc:3037] T 4f1a21869b924bc79a6f82d5d0952dd7 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.234863 8855 raft_consensus.cc:2454] T 4f1a21869b924bc79a6f82d5d0952dd7 P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 1.
I20240502 13:59:45.234884 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1a21869b924bc79a6f82d5d0952dd7" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541" is_pre_election: true
I20240502 13:59:45.235040 9119 raft_consensus.cc:2454] T 4f1a21869b924bc79a6f82d5d0952dd7 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 0.
I20240502 13:59:45.235030 9118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1a21869b924bc79a6f82d5d0952dd7" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541"
I20240502 13:59:45.235165 9317 leader_election.cc:302] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 379b5e0287244d6b94265ccffa51fc05, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:45.235188 9118 raft_consensus.cc:3037] T 4f1a21869b924bc79a6f82d5d0952dd7 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.235302 9547 raft_consensus.cc:2781] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:45.235378 9547 raft_consensus.cc:686] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 LEADER]: Becoming Leader. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Running, Role: LEADER
I20240502 13:59:45.235512 9547 consensus_queue.cc:237] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.236172 9118 raft_consensus.cc:2454] T 4f1a21869b924bc79a6f82d5d0952dd7 P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 1.
I20240502 13:59:45.237124 8578 catalog_manager.cc:5484] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134). New cstate: current_term: 1 leader_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:45.338744 9547 raft_consensus.cc:482] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:45.338909 9547 raft_consensus.cc:504] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.339474 9547 leader_election.cc:288] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939)
I20240502 13:59:45.339936 9118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf2b394ee794547824377bc3a4eae9a" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541" is_pre_election: true
I20240502 13:59:45.340025 8855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf2b394ee794547824377bc3a4eae9a" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05" is_pre_election: true
I20240502 13:59:45.340097 9118 raft_consensus.cc:2454] T fcf2b394ee794547824377bc3a4eae9a P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 0.
I20240502 13:59:45.340160 8855 raft_consensus.cc:2454] T fcf2b394ee794547824377bc3a4eae9a P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 0.
I20240502 13:59:45.340407 9317 leader_election.cc:302] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 379b5e0287244d6b94265ccffa51fc05, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:45.340642 9547 raft_consensus.cc:2781] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:45.340732 9547 raft_consensus.cc:482] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:45.340790 9547 raft_consensus.cc:3037] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.341663 9547 raft_consensus.cc:504] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.341953 9547 leader_election.cc:288] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 election: Requested vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939)
I20240502 13:59:45.342330 9118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf2b394ee794547824377bc3a4eae9a" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541"
I20240502 13:59:45.342375 8855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcf2b394ee794547824377bc3a4eae9a" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05"
I20240502 13:59:45.342449 9118 raft_consensus.cc:3037] T fcf2b394ee794547824377bc3a4eae9a P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.342473 8855 raft_consensus.cc:3037] T fcf2b394ee794547824377bc3a4eae9a P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.343390 9118 raft_consensus.cc:2454] T fcf2b394ee794547824377bc3a4eae9a P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 1.
I20240502 13:59:45.343391 8855 raft_consensus.cc:2454] T fcf2b394ee794547824377bc3a4eae9a P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 1.
I20240502 13:59:45.343680 9317 leader_election.cc:302] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:45.343843 9547 raft_consensus.cc:2781] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:45.343930 9547 raft_consensus.cc:686] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 LEADER]: Becoming Leader. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Running, Role: LEADER
I20240502 13:59:45.344064 9547 consensus_queue.cc:237] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.347708 8578 catalog_manager.cc:5484] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134). New cstate: current_term: 1 leader_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:45.469686 9576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:45.469952 9576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:45.473075 9576 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 13:59:45.593245 8723 raft_consensus.cc:1261] T 3a034d01e1b5402681c95d135646b88f P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Refusing update from remote peer 66ef9d15c0bd46fbb23d5ba11a085ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:45.593614 9482 consensus_queue.cc:1035] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:45.594058 9251 raft_consensus.cc:1261] T 3a034d01e1b5402681c95d135646b88f P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Refusing update from remote peer 66ef9d15c0bd46fbb23d5ba11a085ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:45.594944 8855 raft_consensus.cc:1261] T 4f1a21869b924bc79a6f82d5d0952dd7 P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Refusing update from remote peer 833c2ad4636d4d8e9861339a4ad9e80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:45.595063 9118 raft_consensus.cc:1261] T 4f1a21869b924bc79a6f82d5d0952dd7 P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Refusing update from remote peer 833c2ad4636d4d8e9861339a4ad9e80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:45.595424 9547 consensus_queue.cc:1035] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:45.595543 9478 consensus_queue.cc:1035] T 4f1a21869b924bc79a6f82d5d0952dd7 P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:45.599512 9482 consensus_queue.cc:1035] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:45.600654 8854 raft_consensus.cc:1261] T fcf2b394ee794547824377bc3a4eae9a P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Refusing update from remote peer 833c2ad4636d4d8e9861339a4ad9e80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:45.601016 9119 raft_consensus.cc:1261] T fcf2b394ee794547824377bc3a4eae9a P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Refusing update from remote peer 833c2ad4636d4d8e9861339a4ad9e80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:45.601029 9547 consensus_queue.cc:1035] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:45.601385 9547 consensus_queue.cc:1035] T fcf2b394ee794547824377bc3a4eae9a P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:45.605366 9607 mvcc.cc:205] Tried to move back new op lower bound from 7023240747386343424 to 7023240745925996544. Current Snapshot: MvccSnapshot[applied={T|T < 7023240747386343424}]
I20240502 13:59:45.621305 9609 mvcc.cc:205] Tried to move back new op lower bound from 7023240747386548224 to 7023240745802977280. Current Snapshot: MvccSnapshot[applied={T|T < 7023240747386548224}]
I20240502 13:59:45.626847 9612 mvcc.cc:205] Tried to move back new op lower bound from 7023240747386343424 to 7023240745925996544. Current Snapshot: MvccSnapshot[applied={T|T < 7023240747386343424 or (T in {7023240747386343424})}]
INSERT report
rows total: 2000
time total: 141.966 ms
time per row: 0.0709832 ms
I20240502 13:59:45.661720 8574 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54006:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:45.667011 8703 tablet_service.cc:1467] Processing CreateTablet for tablet e1f07eacd9af4363a0506ad9ebace9fd (DEFAULT_TABLE table=rebalance_test_table_2 [id=c49dbdbc34f34b02ba51928fe15864b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.667126 8965 tablet_service.cc:1467] Processing CreateTablet for tablet e1f07eacd9af4363a0506ad9ebace9fd (DEFAULT_TABLE table=rebalance_test_table_2 [id=c49dbdbc34f34b02ba51928fe15864b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.667358 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f07eacd9af4363a0506ad9ebace9fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.667419 8965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f07eacd9af4363a0506ad9ebace9fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.669222 9637 tablet_bootstrap.cc:492] T e1f07eacd9af4363a0506ad9ebace9fd P 4397244002fd4e0f9246b55727b536fe: Bootstrap starting.
I20240502 13:59:45.669291 9638 tablet_bootstrap.cc:492] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6: Bootstrap starting.
I20240502 13:59:45.669835 9098 tablet_service.cc:1467] Processing CreateTablet for tablet bdae32d952674a2796d571db6b00f839 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c49dbdbc34f34b02ba51928fe15864b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.669952 8834 tablet_service.cc:1467] Processing CreateTablet for tablet bdae32d952674a2796d571db6b00f839 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c49dbdbc34f34b02ba51928fe15864b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.670087 9098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdae32d952674a2796d571db6b00f839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.670100 9637 tablet_bootstrap.cc:654] T e1f07eacd9af4363a0506ad9ebace9fd P 4397244002fd4e0f9246b55727b536fe: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.670193 9638 tablet_bootstrap.cc:654] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.670248 8834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdae32d952674a2796d571db6b00f839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.671270 9638 tablet_bootstrap.cc:492] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6: No bootstrap required, opened a new log
I20240502 13:59:45.671343 9638 ts_tablet_manager.cc:1367] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:45.671754 9638 raft_consensus.cc:348] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:45.671876 9638 raft_consensus.cc:374] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.671919 9638 raft_consensus.cc:729] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.672071 9641 tablet_bootstrap.cc:492] T bdae32d952674a2796d571db6b00f839 P 379b5e0287244d6b94265ccffa51fc05: Bootstrap starting.
I20240502 13:59:45.672102 9638 consensus_queue.cc:260] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:45.672446 9638 ts_tablet_manager.cc:1398] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:45.673099 8965 tablet_service.cc:1467] Processing CreateTablet for tablet 3333dcab7e814b42ae88630f83f6ed84 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c49dbdbc34f34b02ba51928fe15864b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.673173 9231 tablet_service.cc:1467] Processing CreateTablet for tablet e1f07eacd9af4363a0506ad9ebace9fd (DEFAULT_TABLE table=rebalance_test_table_2 [id=c49dbdbc34f34b02ba51928fe15864b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.673322 8965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3333dcab7e814b42ae88630f83f6ed84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.673399 9641 tablet_bootstrap.cc:654] T bdae32d952674a2796d571db6b00f839 P 379b5e0287244d6b94265ccffa51fc05: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.673414 9231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f07eacd9af4363a0506ad9ebace9fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.673894 8703 tablet_service.cc:1467] Processing CreateTablet for tablet 3333dcab7e814b42ae88630f83f6ed84 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c49dbdbc34f34b02ba51928fe15864b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.674096 9642 tablet_bootstrap.cc:492] T bdae32d952674a2796d571db6b00f839 P 2695a61796d043aa823fe3ee966a7541: Bootstrap starting.
I20240502 13:59:45.674142 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3333dcab7e814b42ae88630f83f6ed84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.674535 9230 tablet_service.cc:1467] Processing CreateTablet for tablet 3333dcab7e814b42ae88630f83f6ed84 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c49dbdbc34f34b02ba51928fe15864b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.674746 9230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3333dcab7e814b42ae88630f83f6ed84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.675001 9642 tablet_bootstrap.cc:654] T bdae32d952674a2796d571db6b00f839 P 2695a61796d043aa823fe3ee966a7541: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.676677 9642 tablet_bootstrap.cc:492] T bdae32d952674a2796d571db6b00f839 P 2695a61796d043aa823fe3ee966a7541: No bootstrap required, opened a new log
I20240502 13:59:45.676755 9642 ts_tablet_manager.cc:1367] T bdae32d952674a2796d571db6b00f839 P 2695a61796d043aa823fe3ee966a7541: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 13:59:45.677057 9363 tablet_service.cc:1467] Processing CreateTablet for tablet bdae32d952674a2796d571db6b00f839 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c49dbdbc34f34b02ba51928fe15864b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:45.677098 9642 raft_consensus.cc:348] T bdae32d952674a2796d571db6b00f839 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.677176 9642 raft_consensus.cc:374] T bdae32d952674a2796d571db6b00f839 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.677196 9642 raft_consensus.cc:729] T bdae32d952674a2796d571db6b00f839 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.677286 9363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdae32d952674a2796d571db6b00f839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:45.677294 9642 consensus_queue.cc:260] T bdae32d952674a2796d571db6b00f839 P 2695a61796d043aa823fe3ee966a7541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.677521 9642 ts_tablet_manager.cc:1398] T bdae32d952674a2796d571db6b00f839 P 2695a61796d043aa823fe3ee966a7541: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:45.677996 9637 tablet_bootstrap.cc:492] T e1f07eacd9af4363a0506ad9ebace9fd P 4397244002fd4e0f9246b55727b536fe: No bootstrap required, opened a new log
I20240502 13:59:45.678074 9637 ts_tablet_manager.cc:1367] T e1f07eacd9af4363a0506ad9ebace9fd P 4397244002fd4e0f9246b55727b536fe: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240502 13:59:45.678454 9637 raft_consensus.cc:348] T e1f07eacd9af4363a0506ad9ebace9fd P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:45.678534 9637 raft_consensus.cc:374] T e1f07eacd9af4363a0506ad9ebace9fd P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.678553 9637 raft_consensus.cc:729] T e1f07eacd9af4363a0506ad9ebace9fd P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.678663 9637 consensus_queue.cc:260] T e1f07eacd9af4363a0506ad9ebace9fd P 4397244002fd4e0f9246b55727b536fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:45.678892 9637 ts_tablet_manager.cc:1398] T e1f07eacd9af4363a0506ad9ebace9fd P 4397244002fd4e0f9246b55727b536fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:45.679028 9637 tablet_bootstrap.cc:492] T 3333dcab7e814b42ae88630f83f6ed84 P 4397244002fd4e0f9246b55727b536fe: Bootstrap starting.
I20240502 13:59:45.679714 9637 tablet_bootstrap.cc:654] T 3333dcab7e814b42ae88630f83f6ed84 P 4397244002fd4e0f9246b55727b536fe: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.680781 9638 tablet_bootstrap.cc:492] T 3333dcab7e814b42ae88630f83f6ed84 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Bootstrap starting.
I20240502 13:59:45.681715 9638 tablet_bootstrap.cc:654] T 3333dcab7e814b42ae88630f83f6ed84 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.682525 9646 tablet_bootstrap.cc:492] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e: Bootstrap starting.
I20240502 13:59:45.682593 9637 tablet_bootstrap.cc:492] T 3333dcab7e814b42ae88630f83f6ed84 P 4397244002fd4e0f9246b55727b536fe: No bootstrap required, opened a new log
I20240502 13:59:45.682658 9637 ts_tablet_manager.cc:1367] T 3333dcab7e814b42ae88630f83f6ed84 P 4397244002fd4e0f9246b55727b536fe: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 13:59:45.682891 9638 tablet_bootstrap.cc:492] T 3333dcab7e814b42ae88630f83f6ed84 P 66ef9d15c0bd46fbb23d5ba11a085ba6: No bootstrap required, opened a new log
I20240502 13:59:45.682971 9638 ts_tablet_manager.cc:1367] T 3333dcab7e814b42ae88630f83f6ed84 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:45.683053 9637 raft_consensus.cc:348] T 3333dcab7e814b42ae88630f83f6ed84 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:45.683169 9637 raft_consensus.cc:374] T 3333dcab7e814b42ae88630f83f6ed84 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.683207 9637 raft_consensus.cc:729] T 3333dcab7e814b42ae88630f83f6ed84 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.683331 9637 consensus_queue.cc:260] T 3333dcab7e814b42ae88630f83f6ed84 P 4397244002fd4e0f9246b55727b536fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:45.683403 9646 tablet_bootstrap.cc:654] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.683362 9638 raft_consensus.cc:348] T 3333dcab7e814b42ae88630f83f6ed84 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:45.683439 9638 raft_consensus.cc:374] T 3333dcab7e814b42ae88630f83f6ed84 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.683491 9638 raft_consensus.cc:729] T 3333dcab7e814b42ae88630f83f6ed84 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.683621 9637 ts_tablet_manager.cc:1398] T 3333dcab7e814b42ae88630f83f6ed84 P 4397244002fd4e0f9246b55727b536fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:45.683619 9638 consensus_queue.cc:260] T 3333dcab7e814b42ae88630f83f6ed84 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:45.683874 9638 ts_tablet_manager.cc:1398] T 3333dcab7e814b42ae88630f83f6ed84 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:45.684239 9645 tablet_bootstrap.cc:492] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360: Bootstrap starting.
I20240502 13:59:45.685138 9645 tablet_bootstrap.cc:654] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.686347 9645 tablet_bootstrap.cc:492] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360: No bootstrap required, opened a new log
I20240502 13:59:45.686439 9645 ts_tablet_manager.cc:1367] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:45.686812 9645 raft_consensus.cc:348] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:45.686902 9645 raft_consensus.cc:374] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.686934 9645 raft_consensus.cc:729] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.687036 9645 consensus_queue.cc:260] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:45.687274 9645 ts_tablet_manager.cc:1398] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:45.687418 9645 tablet_bootstrap.cc:492] T e1f07eacd9af4363a0506ad9ebace9fd P b0a54f7311df4dd2be068d3b15a4a360: Bootstrap starting.
I20240502 13:59:45.688213 9645 tablet_bootstrap.cc:654] T e1f07eacd9af4363a0506ad9ebace9fd P b0a54f7311df4dd2be068d3b15a4a360: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:45.689324 9646 tablet_bootstrap.cc:492] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e: No bootstrap required, opened a new log
I20240502 13:59:45.689414 9646 ts_tablet_manager.cc:1367] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 13:59:45.689824 9646 raft_consensus.cc:348] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.689915 9646 raft_consensus.cc:374] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.689952 9646 raft_consensus.cc:729] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.690088 9646 consensus_queue.cc:260] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.690341 9646 ts_tablet_manager.cc:1398] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:45.690788 9645 tablet_bootstrap.cc:492] T e1f07eacd9af4363a0506ad9ebace9fd P b0a54f7311df4dd2be068d3b15a4a360: No bootstrap required, opened a new log
I20240502 13:59:45.690883 9645 ts_tablet_manager.cc:1367] T e1f07eacd9af4363a0506ad9ebace9fd P b0a54f7311df4dd2be068d3b15a4a360: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 13:59:45.691308 9645 raft_consensus.cc:348] T e1f07eacd9af4363a0506ad9ebace9fd P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:45.691455 9645 raft_consensus.cc:374] T e1f07eacd9af4363a0506ad9ebace9fd P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.691495 9645 raft_consensus.cc:729] T e1f07eacd9af4363a0506ad9ebace9fd P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.691635 9645 consensus_queue.cc:260] T e1f07eacd9af4363a0506ad9ebace9fd P b0a54f7311df4dd2be068d3b15a4a360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:45.691920 9645 ts_tablet_manager.cc:1398] T e1f07eacd9af4363a0506ad9ebace9fd P b0a54f7311df4dd2be068d3b15a4a360: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:45.692692 9641 tablet_bootstrap.cc:492] T bdae32d952674a2796d571db6b00f839 P 379b5e0287244d6b94265ccffa51fc05: No bootstrap required, opened a new log
I20240502 13:59:45.692802 9641 ts_tablet_manager.cc:1367] T bdae32d952674a2796d571db6b00f839 P 379b5e0287244d6b94265ccffa51fc05: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240502 13:59:45.693308 9641 raft_consensus.cc:348] T bdae32d952674a2796d571db6b00f839 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.693416 9641 raft_consensus.cc:374] T bdae32d952674a2796d571db6b00f839 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:45.693459 9641 raft_consensus.cc:729] T bdae32d952674a2796d571db6b00f839 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Initialized, Role: FOLLOWER
I20240502 13:59:45.693590 9641 consensus_queue.cc:260] T bdae32d952674a2796d571db6b00f839 P 379b5e0287244d6b94265ccffa51fc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.693873 9641 ts_tablet_manager.cc:1398] T bdae32d952674a2796d571db6b00f839 P 379b5e0287244d6b94265ccffa51fc05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:45.812861 9547 raft_consensus.cc:482] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:45.812983 9547 raft_consensus.cc:504] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.813366 9547 leader_election.cc:288] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939)
I20240502 13:59:45.813803 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdae32d952674a2796d571db6b00f839" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541" is_pre_election: true
I20240502 13:59:45.813803 8854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdae32d952674a2796d571db6b00f839" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05" is_pre_election: true
I20240502 13:59:45.813941 8854 raft_consensus.cc:2454] T bdae32d952674a2796d571db6b00f839 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 0.
I20240502 13:59:45.813941 9119 raft_consensus.cc:2454] T bdae32d952674a2796d571db6b00f839 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 0.
I20240502 13:59:45.814242 9317 leader_election.cc:302] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 379b5e0287244d6b94265ccffa51fc05, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:45.814435 9547 raft_consensus.cc:2781] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:45.814494 9547 raft_consensus.cc:482] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:45.814550 9547 raft_consensus.cc:3037] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.815315 9547 raft_consensus.cc:504] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.815601 9547 leader_election.cc:288] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 election: Requested vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939)
I20240502 13:59:45.815982 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdae32d952674a2796d571db6b00f839" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541"
I20240502 13:59:45.816000 8854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdae32d952674a2796d571db6b00f839" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05"
I20240502 13:59:45.816102 8854 raft_consensus.cc:3037] T bdae32d952674a2796d571db6b00f839 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.816102 9119 raft_consensus.cc:3037] T bdae32d952674a2796d571db6b00f839 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.816907 8854 raft_consensus.cc:2454] T bdae32d952674a2796d571db6b00f839 P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 1.
I20240502 13:59:45.817075 9119 raft_consensus.cc:2454] T bdae32d952674a2796d571db6b00f839 P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 1.
I20240502 13:59:45.817194 9317 leader_election.cc:302] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 379b5e0287244d6b94265ccffa51fc05, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:45.817492 9547 raft_consensus.cc:2781] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:45.817613 9547 raft_consensus.cc:686] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 LEADER]: Becoming Leader. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Running, Role: LEADER
I20240502 13:59:45.817772 9547 consensus_queue.cc:237] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:45.819373 8576 catalog_manager.cc:5484] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134). New cstate: current_term: 1 leader_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:45.855852 9469 raft_consensus.cc:482] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:45.855995 9469 raft_consensus.cc:504] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:45.856483 9469 leader_election.cc:288] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573), b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699)
I20240502 13:59:45.856985 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f07eacd9af4363a0506ad9ebace9fd" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360" is_pre_election: true
I20240502 13:59:45.857009 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f07eacd9af4363a0506ad9ebace9fd" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe" is_pre_election: true
I20240502 13:59:45.857143 9251 raft_consensus.cc:2454] T e1f07eacd9af4363a0506ad9ebace9fd P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 0.
I20240502 13:59:45.857146 8723 raft_consensus.cc:2454] T e1f07eacd9af4363a0506ad9ebace9fd P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 0.
I20240502 13:59:45.857486 8921 leader_election.cc:302] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, 66ef9d15c0bd46fbb23d5ba11a085ba6; no voters:
I20240502 13:59:45.857656 9469 raft_consensus.cc:2781] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:45.857723 9469 raft_consensus.cc:482] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:45.857751 9469 raft_consensus.cc:3037] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.858595 9469 raft_consensus.cc:504] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:45.858908 9469 leader_election.cc:288] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 election: Requested vote from peers 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573), b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699)
I20240502 13:59:45.859215 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f07eacd9af4363a0506ad9ebace9fd" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe"
I20240502 13:59:45.859221 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f07eacd9af4363a0506ad9ebace9fd" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360"
I20240502 13:59:45.859323 9251 raft_consensus.cc:3037] T e1f07eacd9af4363a0506ad9ebace9fd P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.859339 8723 raft_consensus.cc:3037] T e1f07eacd9af4363a0506ad9ebace9fd P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.860026 8723 raft_consensus.cc:2454] T e1f07eacd9af4363a0506ad9ebace9fd P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 1.
I20240502 13:59:45.860047 9251 raft_consensus.cc:2454] T e1f07eacd9af4363a0506ad9ebace9fd P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 1.
I20240502 13:59:45.860307 8921 leader_election.cc:302] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, 66ef9d15c0bd46fbb23d5ba11a085ba6; no voters:
I20240502 13:59:45.860507 9469 raft_consensus.cc:2781] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:45.860586 9469 raft_consensus.cc:686] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 LEADER]: Becoming Leader. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Running, Role: LEADER
I20240502 13:59:45.860742 9469 consensus_queue.cc:237] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:45.862416 8576 catalog_manager.cc:5484] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131). New cstate: current_term: 1 leader_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:46.003938 9617 raft_consensus.cc:482] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:46.004078 9617 raft_consensus.cc:504] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:46.005149 9617 leader_election.cc:288] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873), 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573)
I20240502 13:59:46.009796 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3333dcab7e814b42ae88630f83f6ed84" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe" is_pre_election: true
I20240502 13:59:46.009795 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3333dcab7e814b42ae88630f83f6ed84" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" is_pre_election: true
I20240502 13:59:46.009927 8987 raft_consensus.cc:2454] T 3333dcab7e814b42ae88630f83f6ed84 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 0.
I20240502 13:59:46.009938 8723 raft_consensus.cc:2454] T 3333dcab7e814b42ae88630f83f6ed84 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 0.
I20240502 13:59:46.010289 9185 leader_election.cc:302] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:46.010540 9617 raft_consensus.cc:2781] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:46.010599 9617 raft_consensus.cc:482] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:46.010651 9617 raft_consensus.cc:3037] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.011430 9617 raft_consensus.cc:504] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:46.011771 9617 leader_election.cc:288] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 election: Requested vote from peers 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873), 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573)
I20240502 13:59:46.012084 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3333dcab7e814b42ae88630f83f6ed84" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6"
I20240502 13:59:46.012192 8987 raft_consensus.cc:3037] T 3333dcab7e814b42ae88630f83f6ed84 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.012203 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3333dcab7e814b42ae88630f83f6ed84" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe"
I20240502 13:59:46.012339 8723 raft_consensus.cc:3037] T 3333dcab7e814b42ae88630f83f6ed84 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.012969 8987 raft_consensus.cc:2454] T 3333dcab7e814b42ae88630f83f6ed84 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 1.
I20240502 13:59:46.013124 8723 raft_consensus.cc:2454] T 3333dcab7e814b42ae88630f83f6ed84 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 1.
I20240502 13:59:46.013269 9186 leader_election.cc:302] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66ef9d15c0bd46fbb23d5ba11a085ba6, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:46.013432 9617 raft_consensus.cc:2781] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:46.013639 9617 raft_consensus.cc:686] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 LEADER]: Becoming Leader. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Running, Role: LEADER
I20240502 13:59:46.013928 9617 consensus_queue.cc:237] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:46.015874 8576 catalog_manager.cc:5484] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133). New cstate: current_term: 1 leader_uuid: "b0a54f7311df4dd2be068d3b15a4a360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:46.141558 9659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:46.141770 9659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:46.144718 9659 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 13:59:46.234730 9251 raft_consensus.cc:1261] T e1f07eacd9af4363a0506ad9ebace9fd P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Refusing update from remote peer 66ef9d15c0bd46fbb23d5ba11a085ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:46.235347 9482 consensus_queue.cc:1035] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:46.237797 8854 raft_consensus.cc:1261] T bdae32d952674a2796d571db6b00f839 P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Refusing update from remote peer 833c2ad4636d4d8e9861339a4ad9e80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:46.238950 9119 raft_consensus.cc:1261] T bdae32d952674a2796d571db6b00f839 P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Refusing update from remote peer 833c2ad4636d4d8e9861339a4ad9e80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:46.239362 8723 raft_consensus.cc:1261] T e1f07eacd9af4363a0506ad9ebace9fd P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Refusing update from remote peer 66ef9d15c0bd46fbb23d5ba11a085ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:46.239806 9482 consensus_queue.cc:1035] T e1f07eacd9af4363a0506ad9ebace9fd P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:46.240031 9547 consensus_queue.cc:1035] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:46.242226 9547 consensus_queue.cc:1035] T bdae32d952674a2796d571db6b00f839 P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:46.250552 8723 raft_consensus.cc:1261] T 3333dcab7e814b42ae88630f83f6ed84 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Refusing update from remote peer b0a54f7311df4dd2be068d3b15a4a360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:46.250552 8987 raft_consensus.cc:1261] T 3333dcab7e814b42ae88630f83f6ed84 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Refusing update from remote peer b0a54f7311df4dd2be068d3b15a4a360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:46.251025 9617 consensus_queue.cc:1035] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:46.253230 9617 consensus_queue.cc:1035] T 3333dcab7e814b42ae88630f83f6ed84 P b0a54f7311df4dd2be068d3b15a4a360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 123.523 ms
time per row: 0.0617617 ms
I20240502 13:59:46.309198 8575 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54006:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:46.314867 9098 tablet_service.cc:1467] Processing CreateTablet for tablet ec4945a457e245c992507afb398d7287 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9d3e7d3d4ccc4fcc8f4d73dfb1475ff2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.315198 9098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec4945a457e245c992507afb398d7287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.315301 8834 tablet_service.cc:1467] Processing CreateTablet for tablet ec4945a457e245c992507afb398d7287 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9d3e7d3d4ccc4fcc8f4d73dfb1475ff2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.315301 8835 tablet_service.cc:1467] Processing CreateTablet for tablet bfe2461e26cf47c489b111af958e9309 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9d3e7d3d4ccc4fcc8f4d73dfb1475ff2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.315622 8834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec4945a457e245c992507afb398d7287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.315575 9363 tablet_service.cc:1467] Processing CreateTablet for tablet ec4945a457e245c992507afb398d7287 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9d3e7d3d4ccc4fcc8f4d73dfb1475ff2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.315775 8835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfe2461e26cf47c489b111af958e9309. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.315833 9363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec4945a457e245c992507afb398d7287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.316179 9099 tablet_service.cc:1467] Processing CreateTablet for tablet bfe2461e26cf47c489b111af958e9309 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9d3e7d3d4ccc4fcc8f4d73dfb1475ff2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.316390 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfe2461e26cf47c489b111af958e9309. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.317608 9726 tablet_bootstrap.cc:492] T ec4945a457e245c992507afb398d7287 P 2695a61796d043aa823fe3ee966a7541: Bootstrap starting.
I20240502 13:59:46.318089 9728 tablet_bootstrap.cc:492] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e: Bootstrap starting.
I20240502 13:59:46.318524 9726 tablet_bootstrap.cc:654] T ec4945a457e245c992507afb398d7287 P 2695a61796d043aa823fe3ee966a7541: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.318971 9728 tablet_bootstrap.cc:654] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.319072 8703 tablet_service.cc:1467] Processing CreateTablet for tablet 58d1f57210e04e1482004fe770479d7e (DEFAULT_TABLE table=rebalance_test_table_3 [id=9d3e7d3d4ccc4fcc8f4d73dfb1475ff2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.319167 8965 tablet_service.cc:1467] Processing CreateTablet for tablet 58d1f57210e04e1482004fe770479d7e (DEFAULT_TABLE table=rebalance_test_table_3 [id=9d3e7d3d4ccc4fcc8f4d73dfb1475ff2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.319341 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58d1f57210e04e1482004fe770479d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.319430 8965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58d1f57210e04e1482004fe770479d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.320554 9727 tablet_bootstrap.cc:492] T bfe2461e26cf47c489b111af958e9309 P 379b5e0287244d6b94265ccffa51fc05: Bootstrap starting.
I20240502 13:59:46.321157 9727 tablet_bootstrap.cc:654] T bfe2461e26cf47c489b111af958e9309 P 379b5e0287244d6b94265ccffa51fc05: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.321591 9731 tablet_bootstrap.cc:492] T 58d1f57210e04e1482004fe770479d7e P 66ef9d15c0bd46fbb23d5ba11a085ba6: Bootstrap starting.
I20240502 13:59:46.321645 9231 tablet_service.cc:1467] Processing CreateTablet for tablet 58d1f57210e04e1482004fe770479d7e (DEFAULT_TABLE table=rebalance_test_table_3 [id=9d3e7d3d4ccc4fcc8f4d73dfb1475ff2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.321906 9231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58d1f57210e04e1482004fe770479d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.322371 9728 tablet_bootstrap.cc:492] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e: No bootstrap required, opened a new log
I20240502 13:59:46.322371 9726 tablet_bootstrap.cc:492] T ec4945a457e245c992507afb398d7287 P 2695a61796d043aa823fe3ee966a7541: No bootstrap required, opened a new log
I20240502 13:59:46.322441 9728 ts_tablet_manager.cc:1367] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 13:59:46.322451 9726 ts_tablet_manager.cc:1367] T ec4945a457e245c992507afb398d7287 P 2695a61796d043aa823fe3ee966a7541: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 13:59:46.322523 9731 tablet_bootstrap.cc:654] T 58d1f57210e04e1482004fe770479d7e P 66ef9d15c0bd46fbb23d5ba11a085ba6: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.322863 9728 raft_consensus.cc:348] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:46.322973 9728 raft_consensus.cc:374] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.322914 9726 raft_consensus.cc:348] T ec4945a457e245c992507afb398d7287 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:46.323002 9728 raft_consensus.cc:729] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.323019 9726 raft_consensus.cc:374] T ec4945a457e245c992507afb398d7287 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.323050 9726 raft_consensus.cc:729] T ec4945a457e245c992507afb398d7287 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.323137 9728 consensus_queue.cc:260] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:46.323201 9732 tablet_bootstrap.cc:492] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe: Bootstrap starting.
I20240502 13:59:46.323168 9726 consensus_queue.cc:260] T ec4945a457e245c992507afb398d7287 P 2695a61796d043aa823fe3ee966a7541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:46.323419 9726 ts_tablet_manager.cc:1398] T ec4945a457e245c992507afb398d7287 P 2695a61796d043aa823fe3ee966a7541: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:46.323463 9728 ts_tablet_manager.cc:1398] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:46.323566 9726 tablet_bootstrap.cc:492] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541: Bootstrap starting.
I20240502 13:59:46.323997 9735 tablet_bootstrap.cc:492] T 58d1f57210e04e1482004fe770479d7e P b0a54f7311df4dd2be068d3b15a4a360: Bootstrap starting.
I20240502 13:59:46.324010 9732 tablet_bootstrap.cc:654] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.324402 9726 tablet_bootstrap.cc:654] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.324838 9735 tablet_bootstrap.cc:654] T 58d1f57210e04e1482004fe770479d7e P b0a54f7311df4dd2be068d3b15a4a360: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.325445 9731 tablet_bootstrap.cc:492] T 58d1f57210e04e1482004fe770479d7e P 66ef9d15c0bd46fbb23d5ba11a085ba6: No bootstrap required, opened a new log
I20240502 13:59:46.325527 9731 ts_tablet_manager.cc:1367] T 58d1f57210e04e1482004fe770479d7e P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 13:59:46.325975 9731 raft_consensus.cc:348] T 58d1f57210e04e1482004fe770479d7e P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.326071 9731 raft_consensus.cc:374] T 58d1f57210e04e1482004fe770479d7e P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.326112 9731 raft_consensus.cc:729] T 58d1f57210e04e1482004fe770479d7e P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.326359 9363 tablet_service.cc:1467] Processing CreateTablet for tablet bfe2461e26cf47c489b111af958e9309 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9d3e7d3d4ccc4fcc8f4d73dfb1475ff2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.326403 9731 consensus_queue.cc:260] T 58d1f57210e04e1482004fe770479d7e P 66ef9d15c0bd46fbb23d5ba11a085ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.326610 9363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfe2461e26cf47c489b111af958e9309. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.326695 9731 ts_tablet_manager.cc:1398] T 58d1f57210e04e1482004fe770479d7e P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:46.327245 9727 tablet_bootstrap.cc:492] T bfe2461e26cf47c489b111af958e9309 P 379b5e0287244d6b94265ccffa51fc05: No bootstrap required, opened a new log
I20240502 13:59:46.327322 9727 ts_tablet_manager.cc:1367] T bfe2461e26cf47c489b111af958e9309 P 379b5e0287244d6b94265ccffa51fc05: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 13:59:46.327764 9727 raft_consensus.cc:348] T bfe2461e26cf47c489b111af958e9309 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } }
I20240502 13:59:46.327862 9727 raft_consensus.cc:374] T bfe2461e26cf47c489b111af958e9309 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.327903 9727 raft_consensus.cc:729] T bfe2461e26cf47c489b111af958e9309 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.328042 9727 consensus_queue.cc:260] T bfe2461e26cf47c489b111af958e9309 P 379b5e0287244d6b94265ccffa51fc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } }
I20240502 13:59:46.328298 9727 ts_tablet_manager.cc:1398] T bfe2461e26cf47c489b111af958e9309 P 379b5e0287244d6b94265ccffa51fc05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:46.328480 9727 tablet_bootstrap.cc:492] T ec4945a457e245c992507afb398d7287 P 379b5e0287244d6b94265ccffa51fc05: Bootstrap starting.
I20240502 13:59:46.328743 9728 tablet_bootstrap.cc:492] T bfe2461e26cf47c489b111af958e9309 P 833c2ad4636d4d8e9861339a4ad9e80e: Bootstrap starting.
I20240502 13:59:46.329365 9727 tablet_bootstrap.cc:654] T ec4945a457e245c992507afb398d7287 P 379b5e0287244d6b94265ccffa51fc05: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.329401 9726 tablet_bootstrap.cc:492] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541: No bootstrap required, opened a new log
I20240502 13:59:46.329493 9726 ts_tablet_manager.cc:1367] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 13:59:46.329664 9728 tablet_bootstrap.cc:654] T bfe2461e26cf47c489b111af958e9309 P 833c2ad4636d4d8e9861339a4ad9e80e: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.329988 9726 raft_consensus.cc:348] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } }
I20240502 13:59:46.330098 9726 raft_consensus.cc:374] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.330150 9726 raft_consensus.cc:729] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.330303 9726 consensus_queue.cc:260] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } }
I20240502 13:59:46.330480 9727 tablet_bootstrap.cc:492] T ec4945a457e245c992507afb398d7287 P 379b5e0287244d6b94265ccffa51fc05: No bootstrap required, opened a new log
I20240502 13:59:46.330529 9726 ts_tablet_manager.cc:1398] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:46.330566 9727 ts_tablet_manager.cc:1367] T ec4945a457e245c992507afb398d7287 P 379b5e0287244d6b94265ccffa51fc05: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:46.330987 9727 raft_consensus.cc:348] T ec4945a457e245c992507afb398d7287 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:46.331074 9727 raft_consensus.cc:374] T ec4945a457e245c992507afb398d7287 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.331104 9727 raft_consensus.cc:729] T ec4945a457e245c992507afb398d7287 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.331218 9727 consensus_queue.cc:260] T ec4945a457e245c992507afb398d7287 P 379b5e0287244d6b94265ccffa51fc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:46.331452 9727 ts_tablet_manager.cc:1398] T ec4945a457e245c992507afb398d7287 P 379b5e0287244d6b94265ccffa51fc05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:46.332773 9735 tablet_bootstrap.cc:492] T 58d1f57210e04e1482004fe770479d7e P b0a54f7311df4dd2be068d3b15a4a360: No bootstrap required, opened a new log
I20240502 13:59:46.332813 9728 tablet_bootstrap.cc:492] T bfe2461e26cf47c489b111af958e9309 P 833c2ad4636d4d8e9861339a4ad9e80e: No bootstrap required, opened a new log
I20240502 13:59:46.332938 9735 ts_tablet_manager.cc:1367] T 58d1f57210e04e1482004fe770479d7e P b0a54f7311df4dd2be068d3b15a4a360: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 13:59:46.332942 9728 ts_tablet_manager.cc:1367] T bfe2461e26cf47c489b111af958e9309 P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 13:59:46.333348 9732 tablet_bootstrap.cc:492] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe: No bootstrap required, opened a new log
I20240502 13:59:46.333353 9728 raft_consensus.cc:348] T bfe2461e26cf47c489b111af958e9309 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } }
I20240502 13:59:46.333441 9728 raft_consensus.cc:374] T bfe2461e26cf47c489b111af958e9309 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.333482 9732 ts_tablet_manager.cc:1367] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 13:59:46.333498 9728 raft_consensus.cc:729] T bfe2461e26cf47c489b111af958e9309 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.333524 9735 raft_consensus.cc:348] T 58d1f57210e04e1482004fe770479d7e P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.333633 9735 raft_consensus.cc:374] T 58d1f57210e04e1482004fe770479d7e P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.333635 9728 consensus_queue.cc:260] T bfe2461e26cf47c489b111af958e9309 P 833c2ad4636d4d8e9861339a4ad9e80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } }
I20240502 13:59:46.333703 9735 raft_consensus.cc:729] T 58d1f57210e04e1482004fe770479d7e P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.333920 9728 ts_tablet_manager.cc:1398] T bfe2461e26cf47c489b111af958e9309 P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:46.333885 9735 consensus_queue.cc:260] T 58d1f57210e04e1482004fe770479d7e P b0a54f7311df4dd2be068d3b15a4a360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.334141 9735 ts_tablet_manager.cc:1398] T 58d1f57210e04e1482004fe770479d7e P b0a54f7311df4dd2be068d3b15a4a360: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:46.334478 9732 raft_consensus.cc:348] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.334591 9732 raft_consensus.cc:374] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.334630 9732 raft_consensus.cc:729] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.334794 9732 consensus_queue.cc:260] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.335153 9732 ts_tablet_manager.cc:1398] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:46.416771 9478 raft_consensus.cc:482] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:46.416889 9478 raft_consensus.cc:504] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:46.417310 9478 leader_election.cc:288] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939)
I20240502 13:59:46.417763 8854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec4945a457e245c992507afb398d7287" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05" is_pre_election: true
I20240502 13:59:46.417745 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec4945a457e245c992507afb398d7287" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541" is_pre_election: true
I20240502 13:59:46.417896 9119 raft_consensus.cc:2454] T ec4945a457e245c992507afb398d7287 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 0.
I20240502 13:59:46.417896 8854 raft_consensus.cc:2454] T ec4945a457e245c992507afb398d7287 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 0.
I20240502 13:59:46.418164 9317 leader_election.cc:302] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:46.418323 9478 raft_consensus.cc:2781] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:46.418380 9478 raft_consensus.cc:482] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:46.418430 9478 raft_consensus.cc:3037] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.419472 9478 raft_consensus.cc:504] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:46.419754 9478 leader_election.cc:288] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 election: Requested vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939)
I20240502 13:59:46.420053 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec4945a457e245c992507afb398d7287" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541"
I20240502 13:59:46.420106 8854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec4945a457e245c992507afb398d7287" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05"
I20240502 13:59:46.420182 9119 raft_consensus.cc:3037] T ec4945a457e245c992507afb398d7287 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.420202 8854 raft_consensus.cc:3037] T ec4945a457e245c992507afb398d7287 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.421070 8854 raft_consensus.cc:2454] T ec4945a457e245c992507afb398d7287 P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 1.
I20240502 13:59:46.421070 9119 raft_consensus.cc:2454] T ec4945a457e245c992507afb398d7287 P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 1.
I20240502 13:59:46.421329 9317 leader_election.cc:302] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:46.421475 9478 raft_consensus.cc:2781] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:46.421559 9478 raft_consensus.cc:686] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 LEADER]: Becoming Leader. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Running, Role: LEADER
I20240502 13:59:46.421694 9478 consensus_queue.cc:237] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:46.423271 8579 catalog_manager.cc:5484] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134). New cstate: current_term: 1 leader_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:46.443293 9700 raft_consensus.cc:482] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:46.443408 9700 raft_consensus.cc:504] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.443878 9700 leader_election.cc:288] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873), b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699)
I20240502 13:59:46.444802 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d1f57210e04e1482004fe770479d7e" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360" is_pre_election: true
I20240502 13:59:46.444957 9251 raft_consensus.cc:2454] T 58d1f57210e04e1482004fe770479d7e P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4397244002fd4e0f9246b55727b536fe in term 0.
I20240502 13:59:46.445358 8658 leader_election.cc:302] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:46.445513 9700 raft_consensus.cc:2781] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:46.445578 9700 raft_consensus.cc:482] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:46.445628 9700 raft_consensus.cc:3037] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.446662 9700 raft_consensus.cc:504] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.447031 9700 leader_election.cc:288] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 election: Requested vote from peers 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873), b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699)
I20240502 13:59:46.447449 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d1f57210e04e1482004fe770479d7e" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360"
I20240502 13:59:46.447556 9251 raft_consensus.cc:3037] T 58d1f57210e04e1482004fe770479d7e P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.447878 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d1f57210e04e1482004fe770479d7e" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" is_pre_election: true
I20240502 13:59:46.447963 8986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d1f57210e04e1482004fe770479d7e" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6"
I20240502 13:59:46.448009 8987 raft_consensus.cc:2454] T 58d1f57210e04e1482004fe770479d7e P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4397244002fd4e0f9246b55727b536fe in term 0.
I20240502 13:59:46.448495 9251 raft_consensus.cc:2454] T 58d1f57210e04e1482004fe770479d7e P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4397244002fd4e0f9246b55727b536fe in term 1.
I20240502 13:59:46.448768 8658 leader_election.cc:302] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, b0a54f7311df4dd2be068d3b15a4a360; no voters: 66ef9d15c0bd46fbb23d5ba11a085ba6
I20240502 13:59:46.448922 9700 raft_consensus.cc:2781] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:46.448999 9700 raft_consensus.cc:686] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe [term 1 LEADER]: Becoming Leader. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Running, Role: LEADER
I20240502 13:59:46.449160 9700 consensus_queue.cc:237] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.450682 8579 catalog_manager.cc:5484] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4397244002fd4e0f9246b55727b536fe (127.8.86.129). New cstate: current_term: 1 leader_uuid: "4397244002fd4e0f9246b55727b536fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:46.509830 9703 raft_consensus.cc:482] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:46.509953 9703 raft_consensus.cc:504] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } }
I20240502 13:59:46.511077 9703 leader_election.cc:288] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134:35071), 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939)
I20240502 13:59:46.516031 8854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfe2461e26cf47c489b111af958e9309" candidate_uuid: "2695a61796d043aa823fe3ee966a7541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05" is_pre_election: true
I20240502 13:59:46.516185 8854 raft_consensus.cc:2454] T bfe2461e26cf47c489b111af958e9309 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2695a61796d043aa823fe3ee966a7541 in term 0.
I20240502 13:59:46.516561 9053 leader_election.cc:302] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 379b5e0287244d6b94265ccffa51fc05; no voters:
I20240502 13:59:46.516790 9703 raft_consensus.cc:2781] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:46.516855 9703 raft_consensus.cc:482] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:46.516888 9703 raft_consensus.cc:3037] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.516892 9383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfe2461e26cf47c489b111af958e9309" candidate_uuid: "2695a61796d043aa823fe3ee966a7541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" is_pre_election: true
I20240502 13:59:46.517100 9383 raft_consensus.cc:2454] T bfe2461e26cf47c489b111af958e9309 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2695a61796d043aa823fe3ee966a7541 in term 0.
I20240502 13:59:46.517886 9703 raft_consensus.cc:504] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } }
I20240502 13:59:46.518191 9703 leader_election.cc:288] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 [CANDIDATE]: Term 1 election: Requested vote from peers 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134:35071), 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939)
I20240502 13:59:46.518463 9383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfe2461e26cf47c489b111af958e9309" candidate_uuid: "2695a61796d043aa823fe3ee966a7541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833c2ad4636d4d8e9861339a4ad9e80e"
I20240502 13:59:46.518473 8854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfe2461e26cf47c489b111af958e9309" candidate_uuid: "2695a61796d043aa823fe3ee966a7541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05"
I20240502 13:59:46.518560 8854 raft_consensus.cc:3037] T bfe2461e26cf47c489b111af958e9309 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.518566 9383 raft_consensus.cc:3037] T bfe2461e26cf47c489b111af958e9309 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.519258 9383 raft_consensus.cc:2454] T bfe2461e26cf47c489b111af958e9309 P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2695a61796d043aa823fe3ee966a7541 in term 1.
I20240502 13:59:46.519322 8854 raft_consensus.cc:2454] T bfe2461e26cf47c489b111af958e9309 P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2695a61796d043aa823fe3ee966a7541 in term 1.
I20240502 13:59:46.519534 9054 leader_election.cc:302] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:46.519719 9703 raft_consensus.cc:2781] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:46.519883 9703 raft_consensus.cc:686] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 [term 1 LEADER]: Becoming Leader. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Running, Role: LEADER
I20240502 13:59:46.520139 9703 consensus_queue.cc:237] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } }
I20240502 13:59:46.521917 8579 catalog_manager.cc:5484] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2695a61796d043aa823fe3ee966a7541 (127.8.86.132). New cstate: current_term: 1 leader_uuid: "2695a61796d043aa823fe3ee966a7541" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } health_report { overall_health: HEALTHY } } }
W20240502 13:59:46.658670 9749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:46.658921 9749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:46.661787 9749 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 13:59:46.733131 8854 raft_consensus.cc:1261] T ec4945a457e245c992507afb398d7287 P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Refusing update from remote peer 833c2ad4636d4d8e9861339a4ad9e80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:46.733664 9119 raft_consensus.cc:1261] T ec4945a457e245c992507afb398d7287 P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Refusing update from remote peer 833c2ad4636d4d8e9861339a4ad9e80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:46.734194 8987 raft_consensus.cc:3037] T 58d1f57210e04e1482004fe770479d7e P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.734339 9478 consensus_queue.cc:1035] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:46.735293 8987 raft_consensus.cc:1261] T 58d1f57210e04e1482004fe770479d7e P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Refusing update from remote peer 4397244002fd4e0f9246b55727b536fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:46.735646 9251 raft_consensus.cc:1261] T 58d1f57210e04e1482004fe770479d7e P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Refusing update from remote peer 4397244002fd4e0f9246b55727b536fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:46.736770 9553 consensus_queue.cc:1035] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:46.737512 9383 raft_consensus.cc:1261] T bfe2461e26cf47c489b111af958e9309 P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Refusing update from remote peer 2695a61796d043aa823fe3ee966a7541: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:46.740166 9547 consensus_queue.cc:1035] T ec4945a457e245c992507afb398d7287 P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:46.742350 9703 consensus_queue.cc:1035] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:46.743250 8854 raft_consensus.cc:1261] T bfe2461e26cf47c489b111af958e9309 P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Refusing update from remote peer 2695a61796d043aa823fe3ee966a7541: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:46.743583 9553 consensus_queue.cc:1035] T 58d1f57210e04e1482004fe770479d7e P 4397244002fd4e0f9246b55727b536fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:46.750000 9703 consensus_queue.cc:1035] T bfe2461e26cf47c489b111af958e9309 P 2695a61796d043aa823fe3ee966a7541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 87.2101 ms
time per row: 0.043605 ms
I20240502 13:59:46.797430 8575 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54006:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:46.802500 8965 tablet_service.cc:1467] Processing CreateTablet for tablet f093a86fa3ea4742bd70344f6f183293 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0836913247d24c4da95b491f9c8b14af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.802500 8703 tablet_service.cc:1467] Processing CreateTablet for tablet f093a86fa3ea4742bd70344f6f183293 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0836913247d24c4da95b491f9c8b14af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.802793 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f093a86fa3ea4742bd70344f6f183293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.802793 8965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f093a86fa3ea4742bd70344f6f183293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.804812 9731 tablet_bootstrap.cc:492] T f093a86fa3ea4742bd70344f6f183293 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Bootstrap starting.
I20240502 13:59:46.804812 9732 tablet_bootstrap.cc:492] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe: Bootstrap starting.
I20240502 13:59:46.805053 8834 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb54b4795434871aa9530fc002f0beb (DEFAULT_TABLE table=rebalance_test_table_4 [id=0836913247d24c4da95b491f9c8b14af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.805230 9099 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb54b4795434871aa9530fc002f0beb (DEFAULT_TABLE table=rebalance_test_table_4 [id=0836913247d24c4da95b491f9c8b14af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.805302 8834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb54b4795434871aa9530fc002f0beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.805474 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb54b4795434871aa9530fc002f0beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.805682 9731 tablet_bootstrap.cc:654] T f093a86fa3ea4742bd70344f6f183293 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.805682 9732 tablet_bootstrap.cc:654] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.806694 8703 tablet_service.cc:1467] Processing CreateTablet for tablet 49e3031937504d989e9a63115d8f9c07 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0836913247d24c4da95b491f9c8b14af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.806915 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49e3031937504d989e9a63115d8f9c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.807231 9231 tablet_service.cc:1467] Processing CreateTablet for tablet f093a86fa3ea4742bd70344f6f183293 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0836913247d24c4da95b491f9c8b14af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.807469 9727 tablet_bootstrap.cc:492] T 9cb54b4795434871aa9530fc002f0beb P 379b5e0287244d6b94265ccffa51fc05: Bootstrap starting.
I20240502 13:59:46.807494 9231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f093a86fa3ea4742bd70344f6f183293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.807618 9230 tablet_service.cc:1467] Processing CreateTablet for tablet 49e3031937504d989e9a63115d8f9c07 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0836913247d24c4da95b491f9c8b14af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.807910 9230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49e3031937504d989e9a63115d8f9c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.808295 9727 tablet_bootstrap.cc:654] T 9cb54b4795434871aa9530fc002f0beb P 379b5e0287244d6b94265ccffa51fc05: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.809465 9726 tablet_bootstrap.cc:492] T 9cb54b4795434871aa9530fc002f0beb P 2695a61796d043aa823fe3ee966a7541: Bootstrap starting.
I20240502 13:59:46.809526 9727 tablet_bootstrap.cc:492] T 9cb54b4795434871aa9530fc002f0beb P 379b5e0287244d6b94265ccffa51fc05: No bootstrap required, opened a new log
I20240502 13:59:46.809617 9727 ts_tablet_manager.cc:1367] T 9cb54b4795434871aa9530fc002f0beb P 379b5e0287244d6b94265ccffa51fc05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:46.810029 9727 raft_consensus.cc:348] T 9cb54b4795434871aa9530fc002f0beb P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:46.810148 9727 raft_consensus.cc:374] T 9cb54b4795434871aa9530fc002f0beb P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.810163 9726 tablet_bootstrap.cc:654] T 9cb54b4795434871aa9530fc002f0beb P 2695a61796d043aa823fe3ee966a7541: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.810180 9727 raft_consensus.cc:729] T 9cb54b4795434871aa9530fc002f0beb P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.810328 9727 consensus_queue.cc:260] T 9cb54b4795434871aa9530fc002f0beb P 379b5e0287244d6b94265ccffa51fc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:46.810622 9727 ts_tablet_manager.cc:1398] T 9cb54b4795434871aa9530fc002f0beb P 379b5e0287244d6b94265ccffa51fc05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:46.810886 9732 tablet_bootstrap.cc:492] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe: No bootstrap required, opened a new log
I20240502 13:59:46.810964 9732 ts_tablet_manager.cc:1367] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 13:59:46.811004 9363 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb54b4795434871aa9530fc002f0beb (DEFAULT_TABLE table=rebalance_test_table_4 [id=0836913247d24c4da95b491f9c8b14af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.811245 9363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cb54b4795434871aa9530fc002f0beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.811422 9732 raft_consensus.cc:348] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.811518 9732 raft_consensus.cc:374] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.811558 9732 raft_consensus.cc:729] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.811679 9732 consensus_queue.cc:260] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.812029 9732 ts_tablet_manager.cc:1398] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:46.813256 9728 tablet_bootstrap.cc:492] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e: Bootstrap starting.
I20240502 13:59:46.813277 9735 tablet_bootstrap.cc:492] T f093a86fa3ea4742bd70344f6f183293 P b0a54f7311df4dd2be068d3b15a4a360: Bootstrap starting.
I20240502 13:59:46.814002 9728 tablet_bootstrap.cc:654] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.814059 9735 tablet_bootstrap.cc:654] T f093a86fa3ea4742bd70344f6f183293 P b0a54f7311df4dd2be068d3b15a4a360: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.814882 8965 tablet_service.cc:1467] Processing CreateTablet for tablet 49e3031937504d989e9a63115d8f9c07 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0836913247d24c4da95b491f9c8b14af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:46.815143 8965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49e3031937504d989e9a63115d8f9c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:46.815305 9728 tablet_bootstrap.cc:492] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e: No bootstrap required, opened a new log
I20240502 13:59:46.815383 9728 ts_tablet_manager.cc:1367] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:46.815717 9728 raft_consensus.cc:348] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:46.815798 9728 raft_consensus.cc:374] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.815829 9728 raft_consensus.cc:729] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.815945 9728 consensus_queue.cc:260] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:46.816193 9728 ts_tablet_manager.cc:1398] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:46.819216 9735 tablet_bootstrap.cc:492] T f093a86fa3ea4742bd70344f6f183293 P b0a54f7311df4dd2be068d3b15a4a360: No bootstrap required, opened a new log
I20240502 13:59:46.819299 9735 ts_tablet_manager.cc:1367] T f093a86fa3ea4742bd70344f6f183293 P b0a54f7311df4dd2be068d3b15a4a360: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 13:59:46.819666 9735 raft_consensus.cc:348] T f093a86fa3ea4742bd70344f6f183293 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.819763 9735 raft_consensus.cc:374] T f093a86fa3ea4742bd70344f6f183293 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.819800 9735 raft_consensus.cc:729] T f093a86fa3ea4742bd70344f6f183293 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.819922 9735 consensus_queue.cc:260] T f093a86fa3ea4742bd70344f6f183293 P b0a54f7311df4dd2be068d3b15a4a360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.820171 9735 ts_tablet_manager.cc:1398] T f093a86fa3ea4742bd70344f6f183293 P b0a54f7311df4dd2be068d3b15a4a360: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:46.820312 9735 tablet_bootstrap.cc:492] T 49e3031937504d989e9a63115d8f9c07 P b0a54f7311df4dd2be068d3b15a4a360: Bootstrap starting.
I20240502 13:59:46.821066 9735 tablet_bootstrap.cc:654] T 49e3031937504d989e9a63115d8f9c07 P b0a54f7311df4dd2be068d3b15a4a360: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.822276 9735 tablet_bootstrap.cc:492] T 49e3031937504d989e9a63115d8f9c07 P b0a54f7311df4dd2be068d3b15a4a360: No bootstrap required, opened a new log
I20240502 13:59:46.822572 9735 ts_tablet_manager.cc:1367] T 49e3031937504d989e9a63115d8f9c07 P b0a54f7311df4dd2be068d3b15a4a360: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:46.823050 9735 raft_consensus.cc:348] T 49e3031937504d989e9a63115d8f9c07 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.823444 9735 raft_consensus.cc:374] T 49e3031937504d989e9a63115d8f9c07 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.823529 9735 raft_consensus.cc:729] T 49e3031937504d989e9a63115d8f9c07 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.823711 9735 consensus_queue.cc:260] T 49e3031937504d989e9a63115d8f9c07 P b0a54f7311df4dd2be068d3b15a4a360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.823879 9732 tablet_bootstrap.cc:492] T 49e3031937504d989e9a63115d8f9c07 P 4397244002fd4e0f9246b55727b536fe: Bootstrap starting.
I20240502 13:59:46.823976 9735 ts_tablet_manager.cc:1398] T 49e3031937504d989e9a63115d8f9c07 P b0a54f7311df4dd2be068d3b15a4a360: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:46.824720 9732 tablet_bootstrap.cc:654] T 49e3031937504d989e9a63115d8f9c07 P 4397244002fd4e0f9246b55727b536fe: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.824862 9726 tablet_bootstrap.cc:492] T 9cb54b4795434871aa9530fc002f0beb P 2695a61796d043aa823fe3ee966a7541: No bootstrap required, opened a new log
I20240502 13:59:46.824935 9726 ts_tablet_manager.cc:1367] T 9cb54b4795434871aa9530fc002f0beb P 2695a61796d043aa823fe3ee966a7541: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.001s
I20240502 13:59:46.825311 9726 raft_consensus.cc:348] T 9cb54b4795434871aa9530fc002f0beb P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:46.825407 9726 raft_consensus.cc:374] T 9cb54b4795434871aa9530fc002f0beb P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.825433 9726 raft_consensus.cc:729] T 9cb54b4795434871aa9530fc002f0beb P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.825539 9726 consensus_queue.cc:260] T 9cb54b4795434871aa9530fc002f0beb P 2695a61796d043aa823fe3ee966a7541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:46.825819 9726 ts_tablet_manager.cc:1398] T 9cb54b4795434871aa9530fc002f0beb P 2695a61796d043aa823fe3ee966a7541: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:46.825874 9732 tablet_bootstrap.cc:492] T 49e3031937504d989e9a63115d8f9c07 P 4397244002fd4e0f9246b55727b536fe: No bootstrap required, opened a new log
I20240502 13:59:46.825930 9732 ts_tablet_manager.cc:1367] T 49e3031937504d989e9a63115d8f9c07 P 4397244002fd4e0f9246b55727b536fe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:46.826272 9732 raft_consensus.cc:348] T 49e3031937504d989e9a63115d8f9c07 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.826443 9732 raft_consensus.cc:374] T 49e3031937504d989e9a63115d8f9c07 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.826481 9732 raft_consensus.cc:729] T 49e3031937504d989e9a63115d8f9c07 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.826606 9732 consensus_queue.cc:260] T 49e3031937504d989e9a63115d8f9c07 P 4397244002fd4e0f9246b55727b536fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.826922 9732 ts_tablet_manager.cc:1398] T 49e3031937504d989e9a63115d8f9c07 P 4397244002fd4e0f9246b55727b536fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:46.827023 9731 tablet_bootstrap.cc:492] T f093a86fa3ea4742bd70344f6f183293 P 66ef9d15c0bd46fbb23d5ba11a085ba6: No bootstrap required, opened a new log
I20240502 13:59:46.827097 9731 ts_tablet_manager.cc:1367] T f093a86fa3ea4742bd70344f6f183293 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.000s
I20240502 13:59:46.827466 9731 raft_consensus.cc:348] T f093a86fa3ea4742bd70344f6f183293 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.827569 9731 raft_consensus.cc:374] T f093a86fa3ea4742bd70344f6f183293 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.827595 9731 raft_consensus.cc:729] T f093a86fa3ea4742bd70344f6f183293 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.827700 9731 consensus_queue.cc:260] T f093a86fa3ea4742bd70344f6f183293 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.827913 9731 ts_tablet_manager.cc:1398] T f093a86fa3ea4742bd70344f6f183293 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:46.828054 9731 tablet_bootstrap.cc:492] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Bootstrap starting.
I20240502 13:59:46.828804 9731 tablet_bootstrap.cc:654] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:46.831658 9731 tablet_bootstrap.cc:492] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6: No bootstrap required, opened a new log
I20240502 13:59:46.831730 9731 ts_tablet_manager.cc:1367] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 13:59:46.832041 9731 raft_consensus.cc:348] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.832144 9731 raft_consensus.cc:374] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:46.832165 9731 raft_consensus.cc:729] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Initialized, Role: FOLLOWER
I20240502 13:59:46.832274 9731 consensus_queue.cc:260] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:46.832477 9731 ts_tablet_manager.cc:1398] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:46.968968 9784 raft_consensus.cc:482] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:46.969106 9784 raft_consensus.cc:504] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:46.969483 9784 leader_election.cc:288] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939)
I20240502 13:59:46.969828 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb54b4795434871aa9530fc002f0beb" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541" is_pre_election: true
I20240502 13:59:46.969877 8855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb54b4795434871aa9530fc002f0beb" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05" is_pre_election: true
I20240502 13:59:46.969982 9119 raft_consensus.cc:2454] T 9cb54b4795434871aa9530fc002f0beb P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 0.
I20240502 13:59:46.969987 8855 raft_consensus.cc:2454] T 9cb54b4795434871aa9530fc002f0beb P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 0.
I20240502 13:59:46.970268 9317 leader_election.cc:302] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 379b5e0287244d6b94265ccffa51fc05, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:46.970440 9784 raft_consensus.cc:2781] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:46.970492 9784 raft_consensus.cc:482] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:46.970535 9784 raft_consensus.cc:3037] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.971364 9784 raft_consensus.cc:504] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:46.971638 9784 leader_election.cc:288] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 election: Requested vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939)
I20240502 13:59:46.971926 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb54b4795434871aa9530fc002f0beb" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541"
I20240502 13:59:46.972016 9119 raft_consensus.cc:3037] T 9cb54b4795434871aa9530fc002f0beb P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.971993 8855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cb54b4795434871aa9530fc002f0beb" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05"
I20240502 13:59:46.972126 8855 raft_consensus.cc:3037] T 9cb54b4795434871aa9530fc002f0beb P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:46.972697 9119 raft_consensus.cc:2454] T 9cb54b4795434871aa9530fc002f0beb P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 1.
I20240502 13:59:46.972852 8855 raft_consensus.cc:2454] T 9cb54b4795434871aa9530fc002f0beb P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 1.
I20240502 13:59:46.972940 9317 leader_election.cc:302] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:46.973121 9784 raft_consensus.cc:2781] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:46.973201 9784 raft_consensus.cc:686] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 LEADER]: Becoming Leader. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Running, Role: LEADER
I20240502 13:59:46.973342 9784 consensus_queue.cc:237] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:46.974936 8576 catalog_manager.cc:5484] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134). New cstate: current_term: 1 leader_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:47.046000 9553 raft_consensus.cc:482] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:47.046145 9553 raft_consensus.cc:504] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.046572 9553 leader_election.cc:288] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873), b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699)
I20240502 13:59:47.046996 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f093a86fa3ea4742bd70344f6f183293" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" is_pre_election: true
I20240502 13:59:47.047134 8987 raft_consensus.cc:2454] T f093a86fa3ea4742bd70344f6f183293 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4397244002fd4e0f9246b55727b536fe in term 0.
I20240502 13:59:47.047102 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f093a86fa3ea4742bd70344f6f183293" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360" is_pre_election: true
I20240502 13:59:47.047268 9251 raft_consensus.cc:2454] T f093a86fa3ea4742bd70344f6f183293 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4397244002fd4e0f9246b55727b536fe in term 0.
I20240502 13:59:47.047475 8658 leader_election.cc:302] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, 66ef9d15c0bd46fbb23d5ba11a085ba6; no voters:
I20240502 13:59:47.047708 9553 raft_consensus.cc:2781] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:47.047767 9553 raft_consensus.cc:482] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:47.047812 9553 raft_consensus.cc:3037] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.048626 9553 raft_consensus.cc:504] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.048929 9553 leader_election.cc:288] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 election: Requested vote from peers 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873), b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699)
I20240502 13:59:47.049264 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f093a86fa3ea4742bd70344f6f183293" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6"
I20240502 13:59:47.049295 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f093a86fa3ea4742bd70344f6f183293" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360"
I20240502 13:59:47.049391 9251 raft_consensus.cc:3037] T f093a86fa3ea4742bd70344f6f183293 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.049391 8987 raft_consensus.cc:3037] T f093a86fa3ea4742bd70344f6f183293 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.050347 8987 raft_consensus.cc:2454] T f093a86fa3ea4742bd70344f6f183293 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4397244002fd4e0f9246b55727b536fe in term 1.
I20240502 13:59:47.050347 9251 raft_consensus.cc:2454] T f093a86fa3ea4742bd70344f6f183293 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4397244002fd4e0f9246b55727b536fe in term 1.
I20240502 13:59:47.050623 8658 leader_election.cc:302] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, 66ef9d15c0bd46fbb23d5ba11a085ba6; no voters:
I20240502 13:59:47.050763 9553 raft_consensus.cc:2781] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:47.050854 9553 raft_consensus.cc:686] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe [term 1 LEADER]: Becoming Leader. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Running, Role: LEADER
I20240502 13:59:47.050997 9553 consensus_queue.cc:237] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.052683 8576 catalog_manager.cc:5484] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4397244002fd4e0f9246b55727b536fe (127.8.86.129). New cstate: current_term: 1 leader_uuid: "4397244002fd4e0f9246b55727b536fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:47.062894 9469 raft_consensus.cc:482] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:47.063005 9469 raft_consensus.cc:504] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.063338 9469 leader_election.cc:288] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573), b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699)
I20240502 13:59:47.063681 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49e3031937504d989e9a63115d8f9c07" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360" is_pre_election: true
I20240502 13:59:47.063799 9251 raft_consensus.cc:2454] T 49e3031937504d989e9a63115d8f9c07 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 0.
I20240502 13:59:47.063814 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49e3031937504d989e9a63115d8f9c07" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe" is_pre_election: true
I20240502 13:59:47.063901 8723 raft_consensus.cc:2454] T 49e3031937504d989e9a63115d8f9c07 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 0.
I20240502 13:59:47.064074 8922 leader_election.cc:302] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66ef9d15c0bd46fbb23d5ba11a085ba6, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:47.064249 9469 raft_consensus.cc:2781] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:47.064304 9469 raft_consensus.cc:482] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:47.064344 9469 raft_consensus.cc:3037] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.065027 9469 raft_consensus.cc:504] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.065317 9469 leader_election.cc:288] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 election: Requested vote from peers 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573), b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699)
I20240502 13:59:47.065565 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49e3031937504d989e9a63115d8f9c07" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe"
I20240502 13:59:47.065640 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49e3031937504d989e9a63115d8f9c07" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360"
I20240502 13:59:47.065699 8723 raft_consensus.cc:3037] T 49e3031937504d989e9a63115d8f9c07 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.065743 9251 raft_consensus.cc:3037] T 49e3031937504d989e9a63115d8f9c07 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.066637 8723 raft_consensus.cc:2454] T 49e3031937504d989e9a63115d8f9c07 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 1.
I20240502 13:59:47.066634 9251 raft_consensus.cc:2454] T 49e3031937504d989e9a63115d8f9c07 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 1.
I20240502 13:59:47.066923 8921 leader_election.cc:302] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, 66ef9d15c0bd46fbb23d5ba11a085ba6; no voters:
I20240502 13:59:47.067068 9469 raft_consensus.cc:2781] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:47.067162 9469 raft_consensus.cc:686] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 LEADER]: Becoming Leader. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Running, Role: LEADER
I20240502 13:59:47.067287 9469 consensus_queue.cc:237] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.068816 8576 catalog_manager.cc:5484] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131). New cstate: current_term: 1 leader_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:47.209384 9807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:47.209636 9807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:47.212657 9807 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 13:59:47.305945 9251 raft_consensus.cc:1261] T f093a86fa3ea4742bd70344f6f183293 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Refusing update from remote peer 4397244002fd4e0f9246b55727b536fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:47.306517 9553 consensus_queue.cc:1035] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:47.308260 8987 raft_consensus.cc:1261] T f093a86fa3ea4742bd70344f6f183293 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Refusing update from remote peer 4397244002fd4e0f9246b55727b536fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:47.308733 9700 consensus_queue.cc:1035] T f093a86fa3ea4742bd70344f6f183293 P 4397244002fd4e0f9246b55727b536fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:47.311630 8855 raft_consensus.cc:1261] T 9cb54b4795434871aa9530fc002f0beb P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Refusing update from remote peer 833c2ad4636d4d8e9861339a4ad9e80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:47.312036 9119 raft_consensus.cc:1261] T 9cb54b4795434871aa9530fc002f0beb P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Refusing update from remote peer 833c2ad4636d4d8e9861339a4ad9e80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:47.312256 9784 consensus_queue.cc:1035] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:47.312409 9547 consensus_queue.cc:1035] T 9cb54b4795434871aa9530fc002f0beb P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:47.330391 9251 raft_consensus.cc:1261] T 49e3031937504d989e9a63115d8f9c07 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Refusing update from remote peer 66ef9d15c0bd46fbb23d5ba11a085ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:47.331315 9469 consensus_queue.cc:1035] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:47.334597 8723 raft_consensus.cc:1261] T 49e3031937504d989e9a63115d8f9c07 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Refusing update from remote peer 66ef9d15c0bd46fbb23d5ba11a085ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:47.335111 9469 consensus_queue.cc:1035] T 49e3031937504d989e9a63115d8f9c07 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 134.773 ms
time per row: 0.0673865 ms
I20240502 13:59:47.389907 8572 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54006:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:47.395464 9099 tablet_service.cc:1467] Processing CreateTablet for tablet b45b34b813a745eb9864caab8e44362e (DEFAULT_TABLE table=rebalance_test_table_5 [id=ea7b0be2978641918ba881337950d83a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:47.395520 8834 tablet_service.cc:1467] Processing CreateTablet for tablet b45b34b813a745eb9864caab8e44362e (DEFAULT_TABLE table=rebalance_test_table_5 [id=ea7b0be2978641918ba881337950d83a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:47.395803 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b45b34b813a745eb9864caab8e44362e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:47.395852 8834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b45b34b813a745eb9864caab8e44362e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:47.396200 9363 tablet_service.cc:1467] Processing CreateTablet for tablet b45b34b813a745eb9864caab8e44362e (DEFAULT_TABLE table=rebalance_test_table_5 [id=ea7b0be2978641918ba881337950d83a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:47.396198 8703 tablet_service.cc:1467] Processing CreateTablet for tablet bfa322f449a64cf69ebafc72bca32faa (DEFAULT_TABLE table=rebalance_test_table_5 [id=ea7b0be2978641918ba881337950d83a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:47.396464 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfa322f449a64cf69ebafc72bca32faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:47.396464 9363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b45b34b813a745eb9864caab8e44362e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:47.398186 8835 tablet_service.cc:1467] Processing CreateTablet for tablet 205cc772d7e44e289670490fb705dff8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ea7b0be2978641918ba881337950d83a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:47.398428 8835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205cc772d7e44e289670490fb705dff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:47.399019 9363 tablet_service.cc:1467] Processing CreateTablet for tablet 205cc772d7e44e289670490fb705dff8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ea7b0be2978641918ba881337950d83a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:47.399070 9230 tablet_service.cc:1467] Processing CreateTablet for tablet bfa322f449a64cf69ebafc72bca32faa (DEFAULT_TABLE table=rebalance_test_table_5 [id=ea7b0be2978641918ba881337950d83a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:47.399238 9363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205cc772d7e44e289670490fb705dff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:47.399302 9230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfa322f449a64cf69ebafc72bca32faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:47.400748 9871 tablet_bootstrap.cc:492] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05: Bootstrap starting.
I20240502 13:59:47.400967 8965 tablet_service.cc:1467] Processing CreateTablet for tablet bfa322f449a64cf69ebafc72bca32faa (DEFAULT_TABLE table=rebalance_test_table_5 [id=ea7b0be2978641918ba881337950d83a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:47.401124 9872 tablet_bootstrap.cc:492] T b45b34b813a745eb9864caab8e44362e P 833c2ad4636d4d8e9861339a4ad9e80e: Bootstrap starting.
I20240502 13:59:47.401244 8965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfa322f449a64cf69ebafc72bca32faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:47.401863 9871 tablet_bootstrap.cc:654] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:47.401890 9872 tablet_bootstrap.cc:654] T b45b34b813a745eb9864caab8e44362e P 833c2ad4636d4d8e9861339a4ad9e80e: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:47.402560 9873 tablet_bootstrap.cc:492] T bfa322f449a64cf69ebafc72bca32faa P 4397244002fd4e0f9246b55727b536fe: Bootstrap starting.
I20240502 13:59:47.402866 9099 tablet_service.cc:1467] Processing CreateTablet for tablet 205cc772d7e44e289670490fb705dff8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ea7b0be2978641918ba881337950d83a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:47.403131 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205cc772d7e44e289670490fb705dff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:47.403878 9871 tablet_bootstrap.cc:492] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05: No bootstrap required, opened a new log
I20240502 13:59:47.403949 9871 ts_tablet_manager.cc:1367] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 13:59:47.404297 9871 raft_consensus.cc:348] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:47.404387 9871 raft_consensus.cc:374] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:47.404408 9871 raft_consensus.cc:729] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Initialized, Role: FOLLOWER
I20240502 13:59:47.404511 9871 consensus_queue.cc:260] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:47.404925 9871 ts_tablet_manager.cc:1398] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:47.405098 9871 tablet_bootstrap.cc:492] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05: Bootstrap starting.
I20240502 13:59:47.405115 9874 tablet_bootstrap.cc:492] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360: Bootstrap starting.
I20240502 13:59:47.405876 9871 tablet_bootstrap.cc:654] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:47.405979 9870 tablet_bootstrap.cc:492] T b45b34b813a745eb9864caab8e44362e P 2695a61796d043aa823fe3ee966a7541: Bootstrap starting.
I20240502 13:59:47.406387 9873 tablet_bootstrap.cc:654] T bfa322f449a64cf69ebafc72bca32faa P 4397244002fd4e0f9246b55727b536fe: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:47.406898 9870 tablet_bootstrap.cc:654] T b45b34b813a745eb9864caab8e44362e P 2695a61796d043aa823fe3ee966a7541: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:47.406898 9871 tablet_bootstrap.cc:492] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05: No bootstrap required, opened a new log
I20240502 13:59:47.406989 9871 ts_tablet_manager.cc:1367] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:47.407239 9877 tablet_bootstrap.cc:492] T bfa322f449a64cf69ebafc72bca32faa P 66ef9d15c0bd46fbb23d5ba11a085ba6: Bootstrap starting.
I20240502 13:59:47.407429 9871 raft_consensus.cc:348] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:47.407541 9871 raft_consensus.cc:374] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:47.407570 9871 raft_consensus.cc:729] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Initialized, Role: FOLLOWER
I20240502 13:59:47.407688 9871 consensus_queue.cc:260] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:47.407920 9871 ts_tablet_manager.cc:1398] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:47.407917 9872 tablet_bootstrap.cc:492] T b45b34b813a745eb9864caab8e44362e P 833c2ad4636d4d8e9861339a4ad9e80e: No bootstrap required, opened a new log
I20240502 13:59:47.408005 9872 ts_tablet_manager.cc:1367] T b45b34b813a745eb9864caab8e44362e P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 13:59:47.408150 9877 tablet_bootstrap.cc:654] T bfa322f449a64cf69ebafc72bca32faa P 66ef9d15c0bd46fbb23d5ba11a085ba6: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:47.408443 9872 raft_consensus.cc:348] T b45b34b813a745eb9864caab8e44362e P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:47.408558 9872 raft_consensus.cc:374] T b45b34b813a745eb9864caab8e44362e P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:47.408619 9872 raft_consensus.cc:729] T b45b34b813a745eb9864caab8e44362e P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Initialized, Role: FOLLOWER
I20240502 13:59:47.408748 9872 consensus_queue.cc:260] T b45b34b813a745eb9864caab8e44362e P 833c2ad4636d4d8e9861339a4ad9e80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:47.408795 9873 tablet_bootstrap.cc:492] T bfa322f449a64cf69ebafc72bca32faa P 4397244002fd4e0f9246b55727b536fe: No bootstrap required, opened a new log
I20240502 13:59:47.408866 9873 ts_tablet_manager.cc:1367] T bfa322f449a64cf69ebafc72bca32faa P 4397244002fd4e0f9246b55727b536fe: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 13:59:47.409013 9872 ts_tablet_manager.cc:1398] T b45b34b813a745eb9864caab8e44362e P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:47.409200 9872 tablet_bootstrap.cc:492] T 205cc772d7e44e289670490fb705dff8 P 833c2ad4636d4d8e9861339a4ad9e80e: Bootstrap starting.
I20240502 13:59:47.409951 9873 raft_consensus.cc:348] T bfa322f449a64cf69ebafc72bca32faa P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.410056 9873 raft_consensus.cc:374] T bfa322f449a64cf69ebafc72bca32faa P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:47.410074 9872 tablet_bootstrap.cc:654] T 205cc772d7e44e289670490fb705dff8 P 833c2ad4636d4d8e9861339a4ad9e80e: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:47.410075 9874 tablet_bootstrap.cc:654] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:47.410121 9873 raft_consensus.cc:729] T bfa322f449a64cf69ebafc72bca32faa P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Initialized, Role: FOLLOWER
I20240502 13:59:47.410280 9873 consensus_queue.cc:260] T bfa322f449a64cf69ebafc72bca32faa P 4397244002fd4e0f9246b55727b536fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.410737 9873 ts_tablet_manager.cc:1398] T bfa322f449a64cf69ebafc72bca32faa P 4397244002fd4e0f9246b55727b536fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:47.411633 9872 tablet_bootstrap.cc:492] T 205cc772d7e44e289670490fb705dff8 P 833c2ad4636d4d8e9861339a4ad9e80e: No bootstrap required, opened a new log
I20240502 13:59:47.411773 9872 ts_tablet_manager.cc:1367] T 205cc772d7e44e289670490fb705dff8 P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 13:59:47.411904 9877 tablet_bootstrap.cc:492] T bfa322f449a64cf69ebafc72bca32faa P 66ef9d15c0bd46fbb23d5ba11a085ba6: No bootstrap required, opened a new log
I20240502 13:59:47.412173 9877 ts_tablet_manager.cc:1367] T bfa322f449a64cf69ebafc72bca32faa P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 13:59:47.412271 9872 raft_consensus.cc:348] T 205cc772d7e44e289670490fb705dff8 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:47.412693 9870 tablet_bootstrap.cc:492] T b45b34b813a745eb9864caab8e44362e P 2695a61796d043aa823fe3ee966a7541: No bootstrap required, opened a new log
I20240502 13:59:47.412817 9870 ts_tablet_manager.cc:1367] T b45b34b813a745eb9864caab8e44362e P 2695a61796d043aa823fe3ee966a7541: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 13:59:47.412420 9872 raft_consensus.cc:374] T 205cc772d7e44e289670490fb705dff8 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:47.412863 9872 raft_consensus.cc:729] T 205cc772d7e44e289670490fb705dff8 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Initialized, Role: FOLLOWER
I20240502 13:59:47.413054 9872 consensus_queue.cc:260] T 205cc772d7e44e289670490fb705dff8 P 833c2ad4636d4d8e9861339a4ad9e80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:47.413205 9877 raft_consensus.cc:348] T bfa322f449a64cf69ebafc72bca32faa P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.413277 9870 raft_consensus.cc:348] T b45b34b813a745eb9864caab8e44362e P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:47.413372 9872 ts_tablet_manager.cc:1398] T 205cc772d7e44e289670490fb705dff8 P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:47.413424 9870 raft_consensus.cc:374] T b45b34b813a745eb9864caab8e44362e P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:47.413384 9877 raft_consensus.cc:374] T bfa322f449a64cf69ebafc72bca32faa P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:47.413477 9877 raft_consensus.cc:729] T bfa322f449a64cf69ebafc72bca32faa P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Initialized, Role: FOLLOWER
I20240502 13:59:47.413499 9870 raft_consensus.cc:729] T b45b34b813a745eb9864caab8e44362e P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Initialized, Role: FOLLOWER
I20240502 13:59:47.413640 9870 consensus_queue.cc:260] T b45b34b813a745eb9864caab8e44362e P 2695a61796d043aa823fe3ee966a7541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:47.413667 9877 consensus_queue.cc:260] T bfa322f449a64cf69ebafc72bca32faa P 66ef9d15c0bd46fbb23d5ba11a085ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.413975 9870 ts_tablet_manager.cc:1398] T b45b34b813a745eb9864caab8e44362e P 2695a61796d043aa823fe3ee966a7541: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:47.414093 9877 ts_tablet_manager.cc:1398] T bfa322f449a64cf69ebafc72bca32faa P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:47.414171 9870 tablet_bootstrap.cc:492] T 205cc772d7e44e289670490fb705dff8 P 2695a61796d043aa823fe3ee966a7541: Bootstrap starting.
I20240502 13:59:47.415155 9874 tablet_bootstrap.cc:492] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360: No bootstrap required, opened a new log
I20240502 13:59:47.415185 9870 tablet_bootstrap.cc:654] T 205cc772d7e44e289670490fb705dff8 P 2695a61796d043aa823fe3ee966a7541: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:47.415310 9874 ts_tablet_manager.cc:1367] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 13:59:47.415856 9874 raft_consensus.cc:348] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.415984 9874 raft_consensus.cc:374] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:47.416049 9874 raft_consensus.cc:729] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Initialized, Role: FOLLOWER
I20240502 13:59:47.416218 9870 tablet_bootstrap.cc:492] T 205cc772d7e44e289670490fb705dff8 P 2695a61796d043aa823fe3ee966a7541: No bootstrap required, opened a new log
I20240502 13:59:47.416184 9874 consensus_queue.cc:260] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.416302 9870 ts_tablet_manager.cc:1367] T 205cc772d7e44e289670490fb705dff8 P 2695a61796d043aa823fe3ee966a7541: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:47.416684 9874 ts_tablet_manager.cc:1398] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:47.416688 9870 raft_consensus.cc:348] T 205cc772d7e44e289670490fb705dff8 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:47.416786 9870 raft_consensus.cc:374] T 205cc772d7e44e289670490fb705dff8 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:47.416821 9870 raft_consensus.cc:729] T 205cc772d7e44e289670490fb705dff8 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Initialized, Role: FOLLOWER
I20240502 13:59:47.416934 9870 consensus_queue.cc:260] T 205cc772d7e44e289670490fb705dff8 P 2695a61796d043aa823fe3ee966a7541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:47.417131 9870 ts_tablet_manager.cc:1398] T 205cc772d7e44e289670490fb705dff8 P 2695a61796d043aa823fe3ee966a7541: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:47.422408 9722 raft_consensus.cc:482] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:47.422493 9722 raft_consensus.cc:504] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.422935 9722 leader_election.cc:288] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873), 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573)
I20240502 13:59:47.423275 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfa322f449a64cf69ebafc72bca32faa" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" is_pre_election: true
I20240502 13:59:47.423285 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfa322f449a64cf69ebafc72bca32faa" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe" is_pre_election: true
I20240502 13:59:47.423388 8987 raft_consensus.cc:2454] T bfa322f449a64cf69ebafc72bca32faa P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 0.
I20240502 13:59:47.423413 8723 raft_consensus.cc:2454] T bfa322f449a64cf69ebafc72bca32faa P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 0.
I20240502 13:59:47.423645 9186 leader_election.cc:302] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66ef9d15c0bd46fbb23d5ba11a085ba6, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:47.423810 9722 raft_consensus.cc:2781] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:47.423878 9722 raft_consensus.cc:482] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:47.423914 9722 raft_consensus.cc:3037] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.424643 9722 raft_consensus.cc:504] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.424947 9722 leader_election.cc:288] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 election: Requested vote from peers 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873), 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573)
I20240502 13:59:47.425251 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfa322f449a64cf69ebafc72bca32faa" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6"
I20240502 13:59:47.425351 8987 raft_consensus.cc:3037] T bfa322f449a64cf69ebafc72bca32faa P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.425352 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfa322f449a64cf69ebafc72bca32faa" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe"
I20240502 13:59:47.425464 8723 raft_consensus.cc:3037] T bfa322f449a64cf69ebafc72bca32faa P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.426177 8723 raft_consensus.cc:2454] T bfa322f449a64cf69ebafc72bca32faa P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 1.
I20240502 13:59:47.426278 8987 raft_consensus.cc:2454] T bfa322f449a64cf69ebafc72bca32faa P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 1.
I20240502 13:59:47.426483 9185 leader_election.cc:302] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:47.426625 9722 raft_consensus.cc:2781] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:47.426791 9722 raft_consensus.cc:686] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 [term 1 LEADER]: Becoming Leader. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Running, Role: LEADER
I20240502 13:59:47.426913 9722 consensus_queue.cc:237] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.428524 8574 catalog_manager.cc:5484] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133). New cstate: current_term: 1 leader_uuid: "b0a54f7311df4dd2be068d3b15a4a360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:47.489079 9859 raft_consensus.cc:482] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:47.489213 9859 raft_consensus.cc:504] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:47.490279 9859 leader_election.cc:288] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134:35071)
I20240502 13:59:47.493990 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b45b34b813a745eb9864caab8e44362e" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541" is_pre_election: true
I20240502 13:59:47.494153 9119 raft_consensus.cc:2454] T b45b34b813a745eb9864caab8e44362e P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 0.
I20240502 13:59:47.494148 9383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b45b34b813a745eb9864caab8e44362e" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" is_pre_election: true
I20240502 13:59:47.494279 9383 raft_consensus.cc:2454] T b45b34b813a745eb9864caab8e44362e P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 0.
I20240502 13:59:47.494596 8789 leader_election.cc:302] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 379b5e0287244d6b94265ccffa51fc05; no voters:
I20240502 13:59:47.494851 9859 raft_consensus.cc:2781] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:47.494928 9859 raft_consensus.cc:482] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:47.495002 9859 raft_consensus.cc:3037] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.496021 9859 raft_consensus.cc:504] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:47.496371 9859 leader_election.cc:288] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 election: Requested vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134:35071)
I20240502 13:59:47.496685 9383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b45b34b813a745eb9864caab8e44362e" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833c2ad4636d4d8e9861339a4ad9e80e"
I20240502 13:59:47.496680 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b45b34b813a745eb9864caab8e44362e" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541"
I20240502 13:59:47.496798 9383 raft_consensus.cc:3037] T b45b34b813a745eb9864caab8e44362e P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.496803 9119 raft_consensus.cc:3037] T b45b34b813a745eb9864caab8e44362e P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.497510 9119 raft_consensus.cc:2454] T b45b34b813a745eb9864caab8e44362e P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 1.
I20240502 13:59:47.497586 9383 raft_consensus.cc:2454] T b45b34b813a745eb9864caab8e44362e P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 1.
I20240502 13:59:47.497798 8789 leader_election.cc:302] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 379b5e0287244d6b94265ccffa51fc05; no voters:
I20240502 13:59:47.497957 9859 raft_consensus.cc:2781] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:47.498045 9859 raft_consensus.cc:686] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 [term 1 LEADER]: Becoming Leader. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Running, Role: LEADER
I20240502 13:59:47.498379 9859 consensus_queue.cc:237] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:47.500900 8574 catalog_manager.cc:5484] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "379b5e0287244d6b94265ccffa51fc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:47.520078 9859 raft_consensus.cc:482] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:47.520196 9859 raft_consensus.cc:504] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:47.520622 9859 leader_election.cc:288] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134:35071)
I20240502 13:59:47.521034 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205cc772d7e44e289670490fb705dff8" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541" is_pre_election: true
I20240502 13:59:47.521035 9383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205cc772d7e44e289670490fb705dff8" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" is_pre_election: true
I20240502 13:59:47.521162 9383 raft_consensus.cc:2454] T 205cc772d7e44e289670490fb705dff8 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 0.
I20240502 13:59:47.521168 9119 raft_consensus.cc:2454] T 205cc772d7e44e289670490fb705dff8 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 0.
I20240502 13:59:47.521463 8790 leader_election.cc:302] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 379b5e0287244d6b94265ccffa51fc05, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:47.521721 9859 raft_consensus.cc:2781] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:47.521795 9859 raft_consensus.cc:482] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:47.521847 9859 raft_consensus.cc:3037] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.522660 9859 raft_consensus.cc:504] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:47.522970 9859 leader_election.cc:288] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 election: Requested vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134:35071)
I20240502 13:59:47.523321 9383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205cc772d7e44e289670490fb705dff8" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833c2ad4636d4d8e9861339a4ad9e80e"
I20240502 13:59:47.523321 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205cc772d7e44e289670490fb705dff8" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541"
I20240502 13:59:47.523437 9383 raft_consensus.cc:3037] T 205cc772d7e44e289670490fb705dff8 P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.523437 9119 raft_consensus.cc:3037] T 205cc772d7e44e289670490fb705dff8 P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.524300 9119 raft_consensus.cc:2454] T 205cc772d7e44e289670490fb705dff8 P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 1.
I20240502 13:59:47.524300 9383 raft_consensus.cc:2454] T 205cc772d7e44e289670490fb705dff8 P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 1.
I20240502 13:59:47.524603 8789 leader_election.cc:302] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 379b5e0287244d6b94265ccffa51fc05; no voters:
I20240502 13:59:47.524786 9859 raft_consensus.cc:2781] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:47.524868 9859 raft_consensus.cc:686] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 [term 1 LEADER]: Becoming Leader. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Running, Role: LEADER
I20240502 13:59:47.525030 9859 consensus_queue.cc:237] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:47.527022 8574 catalog_manager.cc:5484] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "379b5e0287244d6b94265ccffa51fc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:47.660663 9892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:47.660935 9892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:47.663882 9892 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 13:59:47.745833 9119 raft_consensus.cc:1261] T b45b34b813a745eb9864caab8e44362e P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Refusing update from remote peer 379b5e0287244d6b94265ccffa51fc05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:47.746941 9383 raft_consensus.cc:1261] T b45b34b813a745eb9864caab8e44362e P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Refusing update from remote peer 379b5e0287244d6b94265ccffa51fc05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:47.747093 9852 consensus_queue.cc:1035] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:47.747381 9859 consensus_queue.cc:1035] T b45b34b813a745eb9864caab8e44362e P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:47.749605 9118 raft_consensus.cc:1261] T 205cc772d7e44e289670490fb705dff8 P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Refusing update from remote peer 379b5e0287244d6b94265ccffa51fc05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:47.751024 9859 consensus_queue.cc:1035] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:47.751859 9382 raft_consensus.cc:1261] T 205cc772d7e44e289670490fb705dff8 P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Refusing update from remote peer 379b5e0287244d6b94265ccffa51fc05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:47.752544 9859 consensus_queue.cc:1035] T 205cc772d7e44e289670490fb705dff8 P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:47.754401 8987 raft_consensus.cc:1261] T bfa322f449a64cf69ebafc72bca32faa P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Refusing update from remote peer b0a54f7311df4dd2be068d3b15a4a360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:47.754789 8723 raft_consensus.cc:1261] T bfa322f449a64cf69ebafc72bca32faa P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Refusing update from remote peer b0a54f7311df4dd2be068d3b15a4a360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:47.755192 9722 consensus_queue.cc:1035] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:47.755935 9722 consensus_queue.cc:1035] T bfa322f449a64cf69ebafc72bca32faa P b0a54f7311df4dd2be068d3b15a4a360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 110.29 ms
time per row: 0.0551449 ms
I20240502 13:59:47.817183 8580 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54006:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:47.822702 9230 tablet_service.cc:1467] Processing CreateTablet for tablet 3836daa0c16c41688432d5cf816eaff9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bc22f1be0fbe429c95073abfebb6c4fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:47.822840 8703 tablet_service.cc:1467] Processing CreateTablet for tablet 3836daa0c16c41688432d5cf816eaff9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bc22f1be0fbe429c95073abfebb6c4fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:47.823029 9230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3836daa0c16c41688432d5cf816eaff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:47.823073 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3836daa0c16c41688432d5cf816eaff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:47.823549 8835 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3778c753514d17965f2f1dbe0a418a (DEFAULT_TABLE table=rebalance_test_table_6 [id=bc22f1be0fbe429c95073abfebb6c4fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:47.823858 8835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a3778c753514d17965f2f1dbe0a418a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:47.825158 9874 tablet_bootstrap.cc:492] T 3836daa0c16c41688432d5cf816eaff9 P b0a54f7311df4dd2be068d3b15a4a360: Bootstrap starting.
I20240502 13:59:47.825356 9099 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3778c753514d17965f2f1dbe0a418a (DEFAULT_TABLE table=rebalance_test_table_6 [id=bc22f1be0fbe429c95073abfebb6c4fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:47.825634 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a3778c753514d17965f2f1dbe0a418a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:47.826009 9874 tablet_bootstrap.cc:654] T 3836daa0c16c41688432d5cf816eaff9 P b0a54f7311df4dd2be068d3b15a4a360: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:47.826463 9873 tablet_bootstrap.cc:492] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe: Bootstrap starting.
I20240502 13:59:47.827371 9873 tablet_bootstrap.cc:654] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:47.827499 9230 tablet_service.cc:1467] Processing CreateTablet for tablet 16ec7874224d4b709d2df8f893f41678 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bc22f1be0fbe429c95073abfebb6c4fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:47.827580 9870 tablet_bootstrap.cc:492] T 8a3778c753514d17965f2f1dbe0a418a P 2695a61796d043aa823fe3ee966a7541: Bootstrap starting.
I20240502 13:59:47.827747 9230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ec7874224d4b709d2df8f893f41678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:47.827934 9871 tablet_bootstrap.cc:492] T 8a3778c753514d17965f2f1dbe0a418a P 379b5e0287244d6b94265ccffa51fc05: Bootstrap starting.
I20240502 13:59:47.828383 9870 tablet_bootstrap.cc:654] T 8a3778c753514d17965f2f1dbe0a418a P 2695a61796d043aa823fe3ee966a7541: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:47.828562 9871 tablet_bootstrap.cc:654] T 8a3778c753514d17965f2f1dbe0a418a P 379b5e0287244d6b94265ccffa51fc05: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:47.829244 8703 tablet_service.cc:1467] Processing CreateTablet for tablet 16ec7874224d4b709d2df8f893f41678 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bc22f1be0fbe429c95073abfebb6c4fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:47.829486 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ec7874224d4b709d2df8f893f41678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:47.829461 8965 tablet_service.cc:1467] Processing CreateTablet for tablet 3836daa0c16c41688432d5cf816eaff9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bc22f1be0fbe429c95073abfebb6c4fe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:47.829785 8965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3836daa0c16c41688432d5cf816eaff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:47.829475 8967 tablet_service.cc:1467] Processing CreateTablet for tablet 16ec7874224d4b709d2df8f893f41678 (DEFAULT_TABLE table=rebalance_test_table_6 [id=bc22f1be0fbe429c95073abfebb6c4fe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:47.829970 8967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16ec7874224d4b709d2df8f893f41678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:47.830792 9363 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3778c753514d17965f2f1dbe0a418a (DEFAULT_TABLE table=rebalance_test_table_6 [id=bc22f1be0fbe429c95073abfebb6c4fe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:47.831038 9363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a3778c753514d17965f2f1dbe0a418a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:47.832882 9872 tablet_bootstrap.cc:492] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e: Bootstrap starting.
I20240502 13:59:47.832934 9877 tablet_bootstrap.cc:492] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Bootstrap starting.
I20240502 13:59:47.833757 9877 tablet_bootstrap.cc:654] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:47.833765 9872 tablet_bootstrap.cc:654] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:47.834829 9873 tablet_bootstrap.cc:492] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe: No bootstrap required, opened a new log
I20240502 13:59:47.834926 9873 ts_tablet_manager.cc:1367] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 13:59:47.834977 9870 tablet_bootstrap.cc:492] T 8a3778c753514d17965f2f1dbe0a418a P 2695a61796d043aa823fe3ee966a7541: No bootstrap required, opened a new log
I20240502 13:59:47.835072 9870 ts_tablet_manager.cc:1367] T 8a3778c753514d17965f2f1dbe0a418a P 2695a61796d043aa823fe3ee966a7541: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 13:59:47.835323 9873 raft_consensus.cc:348] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } }
I20240502 13:59:47.835418 9873 raft_consensus.cc:374] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:47.835453 9873 raft_consensus.cc:729] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Initialized, Role: FOLLOWER
I20240502 13:59:47.835498 9870 raft_consensus.cc:348] T 8a3778c753514d17965f2f1dbe0a418a P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:47.835597 9870 raft_consensus.cc:374] T 8a3778c753514d17965f2f1dbe0a418a P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:47.835584 9873 consensus_queue.cc:260] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } }
I20240502 13:59:47.835625 9870 raft_consensus.cc:729] T 8a3778c753514d17965f2f1dbe0a418a P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Initialized, Role: FOLLOWER
I20240502 13:59:47.835736 9870 consensus_queue.cc:260] T 8a3778c753514d17965f2f1dbe0a418a P 2695a61796d043aa823fe3ee966a7541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:47.835842 9873 ts_tablet_manager.cc:1398] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:47.835980 9870 ts_tablet_manager.cc:1398] T 8a3778c753514d17965f2f1dbe0a418a P 2695a61796d043aa823fe3ee966a7541: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:47.836570 9874 tablet_bootstrap.cc:492] T 3836daa0c16c41688432d5cf816eaff9 P b0a54f7311df4dd2be068d3b15a4a360: No bootstrap required, opened a new log
I20240502 13:59:47.836661 9874 ts_tablet_manager.cc:1367] T 3836daa0c16c41688432d5cf816eaff9 P b0a54f7311df4dd2be068d3b15a4a360: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.001s
I20240502 13:59:47.837038 9874 raft_consensus.cc:348] T 3836daa0c16c41688432d5cf816eaff9 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } }
I20240502 13:59:47.837147 9874 raft_consensus.cc:374] T 3836daa0c16c41688432d5cf816eaff9 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:47.837188 9874 raft_consensus.cc:729] T 3836daa0c16c41688432d5cf816eaff9 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Initialized, Role: FOLLOWER
I20240502 13:59:47.837286 9873 tablet_bootstrap.cc:492] T 16ec7874224d4b709d2df8f893f41678 P 4397244002fd4e0f9246b55727b536fe: Bootstrap starting.
I20240502 13:59:47.837304 9874 consensus_queue.cc:260] T 3836daa0c16c41688432d5cf816eaff9 P b0a54f7311df4dd2be068d3b15a4a360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } }
I20240502 13:59:47.837553 9874 ts_tablet_manager.cc:1398] T 3836daa0c16c41688432d5cf816eaff9 P b0a54f7311df4dd2be068d3b15a4a360: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:47.837724 9874 tablet_bootstrap.cc:492] T 16ec7874224d4b709d2df8f893f41678 P b0a54f7311df4dd2be068d3b15a4a360: Bootstrap starting.
I20240502 13:59:47.837883 9877 tablet_bootstrap.cc:492] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6: No bootstrap required, opened a new log
I20240502 13:59:47.837955 9877 ts_tablet_manager.cc:1367] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 13:59:47.838455 9873 tablet_bootstrap.cc:654] T 16ec7874224d4b709d2df8f893f41678 P 4397244002fd4e0f9246b55727b536fe: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:47.838496 9877 raft_consensus.cc:348] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.838589 9877 raft_consensus.cc:374] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:47.838616 9877 raft_consensus.cc:729] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Initialized, Role: FOLLOWER
I20240502 13:59:47.838719 9877 consensus_queue.cc:260] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.838944 9877 ts_tablet_manager.cc:1398] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:47.839076 9877 tablet_bootstrap.cc:492] T 3836daa0c16c41688432d5cf816eaff9 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Bootstrap starting.
I20240502 13:59:47.839880 9877 tablet_bootstrap.cc:654] T 3836daa0c16c41688432d5cf816eaff9 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:47.840974 9877 tablet_bootstrap.cc:492] T 3836daa0c16c41688432d5cf816eaff9 P 66ef9d15c0bd46fbb23d5ba11a085ba6: No bootstrap required, opened a new log
I20240502 13:59:47.841526 9877 ts_tablet_manager.cc:1367] T 3836daa0c16c41688432d5cf816eaff9 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:47.841939 9877 raft_consensus.cc:348] T 3836daa0c16c41688432d5cf816eaff9 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } }
I20240502 13:59:47.842031 9877 raft_consensus.cc:374] T 3836daa0c16c41688432d5cf816eaff9 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:47.842070 9877 raft_consensus.cc:729] T 3836daa0c16c41688432d5cf816eaff9 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Initialized, Role: FOLLOWER
I20240502 13:59:47.842132 9874 tablet_bootstrap.cc:654] T 16ec7874224d4b709d2df8f893f41678 P b0a54f7311df4dd2be068d3b15a4a360: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:47.842272 9877 consensus_queue.cc:260] T 3836daa0c16c41688432d5cf816eaff9 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } }
I20240502 13:59:47.842655 9877 ts_tablet_manager.cc:1398] T 3836daa0c16c41688432d5cf816eaff9 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:47.843842 9871 tablet_bootstrap.cc:492] T 8a3778c753514d17965f2f1dbe0a418a P 379b5e0287244d6b94265ccffa51fc05: No bootstrap required, opened a new log
I20240502 13:59:47.843945 9871 ts_tablet_manager.cc:1367] T 8a3778c753514d17965f2f1dbe0a418a P 379b5e0287244d6b94265ccffa51fc05: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240502 13:59:47.844409 9871 raft_consensus.cc:348] T 8a3778c753514d17965f2f1dbe0a418a P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:47.844532 9871 raft_consensus.cc:374] T 8a3778c753514d17965f2f1dbe0a418a P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:47.844565 9871 raft_consensus.cc:729] T 8a3778c753514d17965f2f1dbe0a418a P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Initialized, Role: FOLLOWER
I20240502 13:59:47.844698 9871 consensus_queue.cc:260] T 8a3778c753514d17965f2f1dbe0a418a P 379b5e0287244d6b94265ccffa51fc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:47.844874 9874 tablet_bootstrap.cc:492] T 16ec7874224d4b709d2df8f893f41678 P b0a54f7311df4dd2be068d3b15a4a360: No bootstrap required, opened a new log
I20240502 13:59:47.844959 9874 ts_tablet_manager.cc:1367] T 16ec7874224d4b709d2df8f893f41678 P b0a54f7311df4dd2be068d3b15a4a360: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 13:59:47.844970 9871 ts_tablet_manager.cc:1398] T 8a3778c753514d17965f2f1dbe0a418a P 379b5e0287244d6b94265ccffa51fc05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:47.845371 9874 raft_consensus.cc:348] T 16ec7874224d4b709d2df8f893f41678 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.845897 9874 raft_consensus.cc:374] T 16ec7874224d4b709d2df8f893f41678 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:47.845980 9874 raft_consensus.cc:729] T 16ec7874224d4b709d2df8f893f41678 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Initialized, Role: FOLLOWER
I20240502 13:59:47.846016 9872 tablet_bootstrap.cc:492] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e: No bootstrap required, opened a new log
I20240502 13:59:47.846110 9872 ts_tablet_manager.cc:1367] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.001s
I20240502 13:59:47.846210 9874 consensus_queue.cc:260] T 16ec7874224d4b709d2df8f893f41678 P b0a54f7311df4dd2be068d3b15a4a360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.846585 9872 raft_consensus.cc:348] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:47.846691 9872 raft_consensus.cc:374] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:47.846722 9872 raft_consensus.cc:729] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Initialized, Role: FOLLOWER
I20240502 13:59:47.846836 9872 consensus_queue.cc:260] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:47.846982 9873 tablet_bootstrap.cc:492] T 16ec7874224d4b709d2df8f893f41678 P 4397244002fd4e0f9246b55727b536fe: No bootstrap required, opened a new log
I20240502 13:59:47.847065 9873 ts_tablet_manager.cc:1367] T 16ec7874224d4b709d2df8f893f41678 P 4397244002fd4e0f9246b55727b536fe: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 13:59:47.847106 9872 ts_tablet_manager.cc:1398] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:47.847404 9874 ts_tablet_manager.cc:1398] T 16ec7874224d4b709d2df8f893f41678 P b0a54f7311df4dd2be068d3b15a4a360: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:47.847532 9873 raft_consensus.cc:348] T 16ec7874224d4b709d2df8f893f41678 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.847692 9873 raft_consensus.cc:374] T 16ec7874224d4b709d2df8f893f41678 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:47.847743 9873 raft_consensus.cc:729] T 16ec7874224d4b709d2df8f893f41678 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Initialized, Role: FOLLOWER
I20240502 13:59:47.847869 9873 consensus_queue.cc:260] T 16ec7874224d4b709d2df8f893f41678 P 4397244002fd4e0f9246b55727b536fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.848140 9873 ts_tablet_manager.cc:1398] T 16ec7874224d4b709d2df8f893f41678 P 4397244002fd4e0f9246b55727b536fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:47.884488 9784 raft_consensus.cc:482] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:47.884598 9784 raft_consensus.cc:504] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:47.884943 9784 leader_election.cc:288] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939)
I20240502 13:59:47.885270 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3778c753514d17965f2f1dbe0a418a" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541" is_pre_election: true
I20240502 13:59:47.885365 8855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3778c753514d17965f2f1dbe0a418a" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05" is_pre_election: true
I20240502 13:59:47.885414 9119 raft_consensus.cc:2454] T 8a3778c753514d17965f2f1dbe0a418a P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 0.
I20240502 13:59:47.885501 8855 raft_consensus.cc:2454] T 8a3778c753514d17965f2f1dbe0a418a P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 0.
I20240502 13:59:47.885674 9317 leader_election.cc:302] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:47.885862 9784 raft_consensus.cc:2781] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:47.885915 9784 raft_consensus.cc:482] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:47.885972 9784 raft_consensus.cc:3037] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.886798 9784 raft_consensus.cc:504] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:47.887075 9784 leader_election.cc:288] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 election: Requested vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939)
I20240502 13:59:47.887338 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3778c753514d17965f2f1dbe0a418a" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541"
I20240502 13:59:47.887449 9119 raft_consensus.cc:3037] T 8a3778c753514d17965f2f1dbe0a418a P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.887409 8855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3778c753514d17965f2f1dbe0a418a" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05"
I20240502 13:59:47.887552 8855 raft_consensus.cc:3037] T 8a3778c753514d17965f2f1dbe0a418a P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.888422 9119 raft_consensus.cc:2454] T 8a3778c753514d17965f2f1dbe0a418a P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 1.
I20240502 13:59:47.888540 8855 raft_consensus.cc:2454] T 8a3778c753514d17965f2f1dbe0a418a P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 1.
I20240502 13:59:47.888729 9317 leader_election.cc:302] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:47.888873 9784 raft_consensus.cc:2781] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:47.889050 9784 raft_consensus.cc:686] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 LEADER]: Becoming Leader. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Running, Role: LEADER
I20240502 13:59:47.889170 9784 consensus_queue.cc:237] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:47.890802 8572 catalog_manager.cc:5484] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134). New cstate: current_term: 1 leader_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:47.944767 9869 raft_consensus.cc:482] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:47.944903 9869 raft_consensus.cc:504] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.945278 9869 leader_election.cc:288] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573), b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699)
I20240502 13:59:47.945645 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ec7874224d4b709d2df8f893f41678" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe" is_pre_election: true
I20240502 13:59:47.945657 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ec7874224d4b709d2df8f893f41678" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360" is_pre_election: true
I20240502 13:59:47.945775 8723 raft_consensus.cc:2454] T 16ec7874224d4b709d2df8f893f41678 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 0.
I20240502 13:59:47.945785 9251 raft_consensus.cc:2454] T 16ec7874224d4b709d2df8f893f41678 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 0.
I20240502 13:59:47.946046 8921 leader_election.cc:302] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, 66ef9d15c0bd46fbb23d5ba11a085ba6; no voters:
I20240502 13:59:47.946199 9869 raft_consensus.cc:2781] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:47.946260 9869 raft_consensus.cc:482] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:47.946332 9869 raft_consensus.cc:3037] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.947113 9869 raft_consensus.cc:504] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.947418 9869 leader_election.cc:288] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 election: Requested vote from peers 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573), b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699)
I20240502 13:59:47.947744 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ec7874224d4b709d2df8f893f41678" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe"
I20240502 13:59:47.947778 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16ec7874224d4b709d2df8f893f41678" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360"
I20240502 13:59:47.947846 8723 raft_consensus.cc:3037] T 16ec7874224d4b709d2df8f893f41678 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.947896 9251 raft_consensus.cc:3037] T 16ec7874224d4b709d2df8f893f41678 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:47.948544 8723 raft_consensus.cc:2454] T 16ec7874224d4b709d2df8f893f41678 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 1.
I20240502 13:59:47.948683 9251 raft_consensus.cc:2454] T 16ec7874224d4b709d2df8f893f41678 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 1.
I20240502 13:59:47.948781 8921 leader_election.cc:302] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, 66ef9d15c0bd46fbb23d5ba11a085ba6; no voters:
I20240502 13:59:47.948993 9869 raft_consensus.cc:2781] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:47.949144 9869 raft_consensus.cc:686] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 LEADER]: Becoming Leader. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Running, Role: LEADER
I20240502 13:59:47.949307 9869 consensus_queue.cc:237] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } }
I20240502 13:59:47.950911 8572 catalog_manager.cc:5484] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131). New cstate: current_term: 1 leader_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:48.033079 9553 raft_consensus.cc:482] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:48.033222 9553 raft_consensus.cc:504] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } }
I20240502 13:59:48.033679 9553 leader_election.cc:288] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699), 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873)
I20240502 13:59:48.034132 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3836daa0c16c41688432d5cf816eaff9" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360" is_pre_election: true
I20240502 13:59:48.034185 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3836daa0c16c41688432d5cf816eaff9" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" is_pre_election: true
I20240502 13:59:48.034307 9251 raft_consensus.cc:2454] T 3836daa0c16c41688432d5cf816eaff9 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4397244002fd4e0f9246b55727b536fe in term 0.
I20240502 13:59:48.034370 8987 raft_consensus.cc:2454] T 3836daa0c16c41688432d5cf816eaff9 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4397244002fd4e0f9246b55727b536fe in term 0.
I20240502 13:59:48.034623 8658 leader_election.cc:302] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:48.034801 9553 raft_consensus.cc:2781] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:48.034862 9553 raft_consensus.cc:482] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:48.034937 9553 raft_consensus.cc:3037] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:48.035741 9553 raft_consensus.cc:504] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } }
I20240502 13:59:48.036043 9553 leader_election.cc:288] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 election: Requested vote from peers b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699), 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873)
I20240502 13:59:48.036351 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3836daa0c16c41688432d5cf816eaff9" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360"
I20240502 13:59:48.036407 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3836daa0c16c41688432d5cf816eaff9" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6"
I20240502 13:59:48.036453 9251 raft_consensus.cc:3037] T 3836daa0c16c41688432d5cf816eaff9 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:48.036511 8987 raft_consensus.cc:3037] T 3836daa0c16c41688432d5cf816eaff9 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:48.037190 9251 raft_consensus.cc:2454] T 3836daa0c16c41688432d5cf816eaff9 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4397244002fd4e0f9246b55727b536fe in term 1.
I20240502 13:59:48.037226 8987 raft_consensus.cc:2454] T 3836daa0c16c41688432d5cf816eaff9 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4397244002fd4e0f9246b55727b536fe in term 1.
I20240502 13:59:48.037474 8658 leader_election.cc:302] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, 66ef9d15c0bd46fbb23d5ba11a085ba6; no voters:
I20240502 13:59:48.037624 9553 raft_consensus.cc:2781] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:48.037799 9553 raft_consensus.cc:686] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe [term 1 LEADER]: Becoming Leader. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Running, Role: LEADER
I20240502 13:59:48.037941 9553 consensus_queue.cc:237] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } }
I20240502 13:59:48.039590 8572 catalog_manager.cc:5484] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4397244002fd4e0f9246b55727b536fe (127.8.86.129). New cstate: current_term: 1 leader_uuid: "4397244002fd4e0f9246b55727b536fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:48.179504 9947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:48.179759 9947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:48.183877 9947 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 13:59:48.255039 8987 raft_consensus.cc:1261] T 3836daa0c16c41688432d5cf816eaff9 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Refusing update from remote peer 4397244002fd4e0f9246b55727b536fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:48.255373 9251 raft_consensus.cc:1261] T 3836daa0c16c41688432d5cf816eaff9 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Refusing update from remote peer 4397244002fd4e0f9246b55727b536fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:48.255492 9553 consensus_queue.cc:1035] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:48.256080 9553 consensus_queue.cc:1035] T 3836daa0c16c41688432d5cf816eaff9 P 4397244002fd4e0f9246b55727b536fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:48.256971 8855 raft_consensus.cc:1261] T 8a3778c753514d17965f2f1dbe0a418a P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Refusing update from remote peer 833c2ad4636d4d8e9861339a4ad9e80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:48.261113 9941 consensus_queue.cc:1035] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:48.261656 9119 raft_consensus.cc:1261] T 8a3778c753514d17965f2f1dbe0a418a P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Refusing update from remote peer 833c2ad4636d4d8e9861339a4ad9e80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:48.262046 9941 consensus_queue.cc:1035] T 8a3778c753514d17965f2f1dbe0a418a P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:48.264853 8723 raft_consensus.cc:1261] T 16ec7874224d4b709d2df8f893f41678 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Refusing update from remote peer 66ef9d15c0bd46fbb23d5ba11a085ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:48.265200 9250 raft_consensus.cc:1261] T 16ec7874224d4b709d2df8f893f41678 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Refusing update from remote peer 66ef9d15c0bd46fbb23d5ba11a085ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:48.265193 9869 consensus_queue.cc:1035] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:48.266820 9869 consensus_queue.cc:1035] T 16ec7874224d4b709d2df8f893f41678 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 102.059 ms
time per row: 0.0510296 ms
I20240502 13:59:48.327600 8579 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54006:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:48.332536 9363 tablet_service.cc:1467] Processing CreateTablet for tablet 3c45225f7cfa4821a6f75abc2dd0d43d (DEFAULT_TABLE table=rebalance_test_table_7 [id=313484c0e809415e9287fa42a0b79049]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:48.332827 9363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c45225f7cfa4821a6f75abc2dd0d43d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.332825 8835 tablet_service.cc:1467] Processing CreateTablet for tablet 3c45225f7cfa4821a6f75abc2dd0d43d (DEFAULT_TABLE table=rebalance_test_table_7 [id=313484c0e809415e9287fa42a0b79049]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:48.333053 8835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c45225f7cfa4821a6f75abc2dd0d43d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.333124 9099 tablet_service.cc:1467] Processing CreateTablet for tablet 3c45225f7cfa4821a6f75abc2dd0d43d (DEFAULT_TABLE table=rebalance_test_table_7 [id=313484c0e809415e9287fa42a0b79049]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:48.333348 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c45225f7cfa4821a6f75abc2dd0d43d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.334542 8834 tablet_service.cc:1467] Processing CreateTablet for tablet b0012eb13b8b49c4a8ebd0fbf5bc70be (DEFAULT_TABLE table=rebalance_test_table_7 [id=313484c0e809415e9287fa42a0b79049]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:48.334764 8834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0012eb13b8b49c4a8ebd0fbf5bc70be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.335026 9872 tablet_bootstrap.cc:492] T 3c45225f7cfa4821a6f75abc2dd0d43d P 833c2ad4636d4d8e9861339a4ad9e80e: Bootstrap starting.
I20240502 13:59:48.335189 9871 tablet_bootstrap.cc:492] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05: Bootstrap starting.
I20240502 13:59:48.335458 8703 tablet_service.cc:1467] Processing CreateTablet for tablet 16eeb3fe7ac34432b9dce18727289e1c (DEFAULT_TABLE table=rebalance_test_table_7 [id=313484c0e809415e9287fa42a0b79049]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:48.335722 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16eeb3fe7ac34432b9dce18727289e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.335909 9872 tablet_bootstrap.cc:654] T 3c45225f7cfa4821a6f75abc2dd0d43d P 833c2ad4636d4d8e9861339a4ad9e80e: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.335909 9871 tablet_bootstrap.cc:654] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.336943 9872 tablet_bootstrap.cc:492] T 3c45225f7cfa4821a6f75abc2dd0d43d P 833c2ad4636d4d8e9861339a4ad9e80e: No bootstrap required, opened a new log
I20240502 13:59:48.337033 9872 ts_tablet_manager.cc:1367] T 3c45225f7cfa4821a6f75abc2dd0d43d P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:48.337420 9871 tablet_bootstrap.cc:492] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05: No bootstrap required, opened a new log
I20240502 13:59:48.337499 9871 ts_tablet_manager.cc:1367] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:48.337460 9872 raft_consensus.cc:348] T 3c45225f7cfa4821a6f75abc2dd0d43d P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:48.337553 9872 raft_consensus.cc:374] T 3c45225f7cfa4821a6f75abc2dd0d43d P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.337596 9872 raft_consensus.cc:729] T 3c45225f7cfa4821a6f75abc2dd0d43d P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.337702 9872 consensus_queue.cc:260] T 3c45225f7cfa4821a6f75abc2dd0d43d P 833c2ad4636d4d8e9861339a4ad9e80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:48.337869 9871 raft_consensus.cc:348] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:48.337950 9871 raft_consensus.cc:374] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.337944 9870 tablet_bootstrap.cc:492] T 3c45225f7cfa4821a6f75abc2dd0d43d P 2695a61796d043aa823fe3ee966a7541: Bootstrap starting.
I20240502 13:59:48.337971 9872 ts_tablet_manager.cc:1398] T 3c45225f7cfa4821a6f75abc2dd0d43d P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:48.337981 9871 raft_consensus.cc:729] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.338095 9871 consensus_queue.cc:260] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:48.338394 9871 ts_tablet_manager.cc:1398] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:48.338555 9871 tablet_bootstrap.cc:492] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05: Bootstrap starting.
I20240502 13:59:48.338845 9870 tablet_bootstrap.cc:654] T 3c45225f7cfa4821a6f75abc2dd0d43d P 2695a61796d043aa823fe3ee966a7541: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.339207 9871 tablet_bootstrap.cc:654] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.339416 9873 tablet_bootstrap.cc:492] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe: Bootstrap starting.
I20240502 13:59:48.339808 9870 tablet_bootstrap.cc:492] T 3c45225f7cfa4821a6f75abc2dd0d43d P 2695a61796d043aa823fe3ee966a7541: No bootstrap required, opened a new log
I20240502 13:59:48.339824 9363 tablet_service.cc:1467] Processing CreateTablet for tablet b0012eb13b8b49c4a8ebd0fbf5bc70be (DEFAULT_TABLE table=rebalance_test_table_7 [id=313484c0e809415e9287fa42a0b79049]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:48.339884 9870 ts_tablet_manager.cc:1367] T 3c45225f7cfa4821a6f75abc2dd0d43d P 2695a61796d043aa823fe3ee966a7541: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:48.340037 9363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0012eb13b8b49c4a8ebd0fbf5bc70be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.340092 9871 tablet_bootstrap.cc:492] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05: No bootstrap required, opened a new log
I20240502 13:59:48.340162 9871 ts_tablet_manager.cc:1367] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:48.340302 9870 raft_consensus.cc:348] T 3c45225f7cfa4821a6f75abc2dd0d43d P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:48.340363 9873 tablet_bootstrap.cc:654] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.340391 9870 raft_consensus.cc:374] T 3c45225f7cfa4821a6f75abc2dd0d43d P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.340418 9870 raft_consensus.cc:729] T 3c45225f7cfa4821a6f75abc2dd0d43d P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.340569 9870 consensus_queue.cc:260] T 3c45225f7cfa4821a6f75abc2dd0d43d P 2695a61796d043aa823fe3ee966a7541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:48.340631 9871 raft_consensus.cc:348] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:48.340739 9871 raft_consensus.cc:374] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.340768 9871 raft_consensus.cc:729] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.340852 9870 ts_tablet_manager.cc:1398] T 3c45225f7cfa4821a6f75abc2dd0d43d P 2695a61796d043aa823fe3ee966a7541: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:48.341055 9871 consensus_queue.cc:260] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:48.341676 9871 ts_tablet_manager.cc:1398] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:48.342317 9099 tablet_service.cc:1467] Processing CreateTablet for tablet b0012eb13b8b49c4a8ebd0fbf5bc70be (DEFAULT_TABLE table=rebalance_test_table_7 [id=313484c0e809415e9287fa42a0b79049]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:48.342242 8965 tablet_service.cc:1467] Processing CreateTablet for tablet 16eeb3fe7ac34432b9dce18727289e1c (DEFAULT_TABLE table=rebalance_test_table_7 [id=313484c0e809415e9287fa42a0b79049]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:48.342518 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0012eb13b8b49c4a8ebd0fbf5bc70be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.342530 8965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16eeb3fe7ac34432b9dce18727289e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.343262 9872 tablet_bootstrap.cc:492] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 833c2ad4636d4d8e9861339a4ad9e80e: Bootstrap starting.
I20240502 13:59:48.344282 9872 tablet_bootstrap.cc:654] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 833c2ad4636d4d8e9861339a4ad9e80e: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.344684 9230 tablet_service.cc:1467] Processing CreateTablet for tablet 16eeb3fe7ac34432b9dce18727289e1c (DEFAULT_TABLE table=rebalance_test_table_7 [id=313484c0e809415e9287fa42a0b79049]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:48.344946 9230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16eeb3fe7ac34432b9dce18727289e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.344954 9870 tablet_bootstrap.cc:492] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 2695a61796d043aa823fe3ee966a7541: Bootstrap starting.
I20240502 13:59:48.345556 9873 tablet_bootstrap.cc:492] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe: No bootstrap required, opened a new log
I20240502 13:59:48.345656 9873 ts_tablet_manager.cc:1367] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 13:59:48.345728 9870 tablet_bootstrap.cc:654] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 2695a61796d043aa823fe3ee966a7541: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.346091 9873 raft_consensus.cc:348] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.346226 9873 raft_consensus.cc:374] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.346263 9873 raft_consensus.cc:729] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.346406 9873 consensus_queue.cc:260] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.346521 9877 tablet_bootstrap.cc:492] T 16eeb3fe7ac34432b9dce18727289e1c P 66ef9d15c0bd46fbb23d5ba11a085ba6: Bootstrap starting.
I20240502 13:59:48.346678 9873 ts_tablet_manager.cc:1398] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:48.346843 9870 tablet_bootstrap.cc:492] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 2695a61796d043aa823fe3ee966a7541: No bootstrap required, opened a new log
I20240502 13:59:48.346904 9870 ts_tablet_manager.cc:1367] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 2695a61796d043aa823fe3ee966a7541: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:48.347270 9870 raft_consensus.cc:348] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:48.347327 9877 tablet_bootstrap.cc:654] T 16eeb3fe7ac34432b9dce18727289e1c P 66ef9d15c0bd46fbb23d5ba11a085ba6: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.347350 9870 raft_consensus.cc:374] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.347376 9870 raft_consensus.cc:729] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.347479 9870 consensus_queue.cc:260] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 2695a61796d043aa823fe3ee966a7541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:48.347748 9870 ts_tablet_manager.cc:1398] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 2695a61796d043aa823fe3ee966a7541: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:48.348556 9877 tablet_bootstrap.cc:492] T 16eeb3fe7ac34432b9dce18727289e1c P 66ef9d15c0bd46fbb23d5ba11a085ba6: No bootstrap required, opened a new log
I20240502 13:59:48.348654 9877 ts_tablet_manager.cc:1367] T 16eeb3fe7ac34432b9dce18727289e1c P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 13:59:48.349041 9877 raft_consensus.cc:348] T 16eeb3fe7ac34432b9dce18727289e1c P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.349153 9877 raft_consensus.cc:374] T 16eeb3fe7ac34432b9dce18727289e1c P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.349184 9877 raft_consensus.cc:729] T 16eeb3fe7ac34432b9dce18727289e1c P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.349325 9877 consensus_queue.cc:260] T 16eeb3fe7ac34432b9dce18727289e1c P 66ef9d15c0bd46fbb23d5ba11a085ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.349646 9877 ts_tablet_manager.cc:1398] T 16eeb3fe7ac34432b9dce18727289e1c P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:48.352118 10006 tablet_bootstrap.cc:492] T 16eeb3fe7ac34432b9dce18727289e1c P b0a54f7311df4dd2be068d3b15a4a360: Bootstrap starting.
I20240502 13:59:48.354387 10006 tablet_bootstrap.cc:654] T 16eeb3fe7ac34432b9dce18727289e1c P b0a54f7311df4dd2be068d3b15a4a360: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.355355 9872 tablet_bootstrap.cc:492] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 833c2ad4636d4d8e9861339a4ad9e80e: No bootstrap required, opened a new log
I20240502 13:59:48.355430 9872 ts_tablet_manager.cc:1367] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240502 13:59:48.355595 10006 tablet_bootstrap.cc:492] T 16eeb3fe7ac34432b9dce18727289e1c P b0a54f7311df4dd2be068d3b15a4a360: No bootstrap required, opened a new log
I20240502 13:59:48.355675 10006 ts_tablet_manager.cc:1367] T 16eeb3fe7ac34432b9dce18727289e1c P b0a54f7311df4dd2be068d3b15a4a360: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 13:59:48.355784 9872 raft_consensus.cc:348] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:48.355865 9872 raft_consensus.cc:374] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.355888 9872 raft_consensus.cc:729] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.355993 9872 consensus_queue.cc:260] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 833c2ad4636d4d8e9861339a4ad9e80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:48.356216 9872 ts_tablet_manager.cc:1398] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:48.356278 10006 raft_consensus.cc:348] T 16eeb3fe7ac34432b9dce18727289e1c P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.356549 10006 raft_consensus.cc:374] T 16eeb3fe7ac34432b9dce18727289e1c P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.356628 10006 raft_consensus.cc:729] T 16eeb3fe7ac34432b9dce18727289e1c P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.356783 10006 consensus_queue.cc:260] T 16eeb3fe7ac34432b9dce18727289e1c P b0a54f7311df4dd2be068d3b15a4a360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.358083 10006 ts_tablet_manager.cc:1398] T 16eeb3fe7ac34432b9dce18727289e1c P b0a54f7311df4dd2be068d3b15a4a360: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:48.384397 10001 raft_consensus.cc:482] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:48.384514 10001 raft_consensus.cc:504] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:48.384894 10001 leader_election.cc:288] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134:35071)
I20240502 13:59:48.385174 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0012eb13b8b49c4a8ebd0fbf5bc70be" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541" is_pre_election: true
I20240502 13:59:48.385197 9383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0012eb13b8b49c4a8ebd0fbf5bc70be" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" is_pre_election: true
I20240502 13:59:48.385284 9119 raft_consensus.cc:2454] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 0.
I20240502 13:59:48.385313 9383 raft_consensus.cc:2454] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 0.
I20240502 13:59:48.385521 8790 leader_election.cc:302] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 379b5e0287244d6b94265ccffa51fc05, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:48.385775 10001 raft_consensus.cc:2781] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:48.385838 10001 raft_consensus.cc:482] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:48.385890 10001 raft_consensus.cc:3037] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:48.386834 10001 raft_consensus.cc:504] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:48.387112 10001 leader_election.cc:288] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 election: Requested vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134:35071)
I20240502 13:59:48.387372 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0012eb13b8b49c4a8ebd0fbf5bc70be" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541"
I20240502 13:59:48.387434 9383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0012eb13b8b49c4a8ebd0fbf5bc70be" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833c2ad4636d4d8e9861339a4ad9e80e"
I20240502 13:59:48.387488 9119 raft_consensus.cc:3037] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:48.387545 9383 raft_consensus.cc:3037] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:48.388355 9383 raft_consensus.cc:2454] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 1.
I20240502 13:59:48.388355 9119 raft_consensus.cc:2454] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 1.
I20240502 13:59:48.388592 8790 leader_election.cc:302] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 379b5e0287244d6b94265ccffa51fc05, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:48.388732 10001 raft_consensus.cc:2781] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:48.388798 10001 raft_consensus.cc:686] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 [term 1 LEADER]: Becoming Leader. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Running, Role: LEADER
I20240502 13:59:48.388908 10001 consensus_queue.cc:237] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:48.390492 8580 catalog_manager.cc:5484] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "379b5e0287244d6b94265ccffa51fc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:48.456647 10001 raft_consensus.cc:482] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:48.456770 10001 raft_consensus.cc:504] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:48.457171 10001 leader_election.cc:288] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134:35071)
I20240502 13:59:48.457486 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c45225f7cfa4821a6f75abc2dd0d43d" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541" is_pre_election: true
I20240502 13:59:48.457509 9383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c45225f7cfa4821a6f75abc2dd0d43d" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" is_pre_election: true
I20240502 13:59:48.457636 9119 raft_consensus.cc:2454] T 3c45225f7cfa4821a6f75abc2dd0d43d P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 0.
I20240502 13:59:48.457649 9383 raft_consensus.cc:2454] T 3c45225f7cfa4821a6f75abc2dd0d43d P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 0.
I20240502 13:59:48.457908 8789 leader_election.cc:302] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 379b5e0287244d6b94265ccffa51fc05; no voters:
I20240502 13:59:48.458070 10001 raft_consensus.cc:2781] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:48.458127 10001 raft_consensus.cc:482] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:48.458178 10001 raft_consensus.cc:3037] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:48.459051 10001 raft_consensus.cc:504] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:48.459363 10001 leader_election.cc:288] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 election: Requested vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134:35071)
I20240502 13:59:48.459674 9383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c45225f7cfa4821a6f75abc2dd0d43d" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833c2ad4636d4d8e9861339a4ad9e80e"
I20240502 13:59:48.459672 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c45225f7cfa4821a6f75abc2dd0d43d" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541"
I20240502 13:59:48.459780 9383 raft_consensus.cc:3037] T 3c45225f7cfa4821a6f75abc2dd0d43d P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:48.459784 9119 raft_consensus.cc:3037] T 3c45225f7cfa4821a6f75abc2dd0d43d P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:48.460453 9119 raft_consensus.cc:2454] T 3c45225f7cfa4821a6f75abc2dd0d43d P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 1.
I20240502 13:59:48.460515 9383 raft_consensus.cc:2454] T 3c45225f7cfa4821a6f75abc2dd0d43d P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 1.
I20240502 13:59:48.460697 8789 leader_election.cc:302] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 379b5e0287244d6b94265ccffa51fc05; no voters:
I20240502 13:59:48.460870 10001 raft_consensus.cc:2781] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:48.460980 10001 raft_consensus.cc:686] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 [term 1 LEADER]: Becoming Leader. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Running, Role: LEADER
I20240502 13:59:48.461133 10001 consensus_queue.cc:237] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:48.462739 8580 catalog_manager.cc:5484] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "379b5e0287244d6b94265ccffa51fc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:48.593880 9945 raft_consensus.cc:482] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:48.594028 9945 raft_consensus.cc:504] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.594451 9945 leader_election.cc:288] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699), 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873)
I20240502 13:59:48.594830 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16eeb3fe7ac34432b9dce18727289e1c" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360" is_pre_election: true
I20240502 13:59:48.594873 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16eeb3fe7ac34432b9dce18727289e1c" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" is_pre_election: true
I20240502 13:59:48.594967 9251 raft_consensus.cc:2454] T 16eeb3fe7ac34432b9dce18727289e1c P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4397244002fd4e0f9246b55727b536fe in term 0.
I20240502 13:59:48.594991 8987 raft_consensus.cc:2454] T 16eeb3fe7ac34432b9dce18727289e1c P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4397244002fd4e0f9246b55727b536fe in term 0.
I20240502 13:59:48.595270 8658 leader_election.cc:302] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:48.595433 9945 raft_consensus.cc:2781] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:48.595485 9945 raft_consensus.cc:482] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:48.595510 9945 raft_consensus.cc:3037] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:48.596408 9945 raft_consensus.cc:504] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.596715 9945 leader_election.cc:288] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 election: Requested vote from peers b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699), 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873)
I20240502 13:59:48.597018 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16eeb3fe7ac34432b9dce18727289e1c" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360"
I20240502 13:59:48.597081 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16eeb3fe7ac34432b9dce18727289e1c" candidate_uuid: "4397244002fd4e0f9246b55727b536fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6"
I20240502 13:59:48.597143 9251 raft_consensus.cc:3037] T 16eeb3fe7ac34432b9dce18727289e1c P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:48.597198 8987 raft_consensus.cc:3037] T 16eeb3fe7ac34432b9dce18727289e1c P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:48.597862 9251 raft_consensus.cc:2454] T 16eeb3fe7ac34432b9dce18727289e1c P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4397244002fd4e0f9246b55727b536fe in term 1.
I20240502 13:59:48.597884 8987 raft_consensus.cc:2454] T 16eeb3fe7ac34432b9dce18727289e1c P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4397244002fd4e0f9246b55727b536fe in term 1.
I20240502 13:59:48.598119 8658 leader_election.cc:302] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, 66ef9d15c0bd46fbb23d5ba11a085ba6; no voters:
I20240502 13:59:48.598281 9945 raft_consensus.cc:2781] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:48.598398 9945 raft_consensus.cc:686] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe [term 1 LEADER]: Becoming Leader. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Running, Role: LEADER
I20240502 13:59:48.598547 9945 consensus_queue.cc:237] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.600148 8580 catalog_manager.cc:5484] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4397244002fd4e0f9246b55727b536fe (127.8.86.129). New cstate: current_term: 1 leader_uuid: "4397244002fd4e0f9246b55727b536fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } health_report { overall_health: HEALTHY } } }
W20240502 13:59:48.734488 10011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:48.734714 10011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:48.737689 10011 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 13:59:48.826681 9383 raft_consensus.cc:1261] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Refusing update from remote peer 379b5e0287244d6b94265ccffa51fc05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:48.826720 9119 raft_consensus.cc:1261] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Refusing update from remote peer 379b5e0287244d6b94265ccffa51fc05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:48.827145 10001 consensus_queue.cc:1035] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:48.827320 10002 consensus_queue.cc:1035] T b0012eb13b8b49c4a8ebd0fbf5bc70be P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:48.832836 9382 raft_consensus.cc:1261] T 3c45225f7cfa4821a6f75abc2dd0d43d P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Refusing update from remote peer 379b5e0287244d6b94265ccffa51fc05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:48.833387 9118 raft_consensus.cc:1261] T 3c45225f7cfa4821a6f75abc2dd0d43d P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Refusing update from remote peer 379b5e0287244d6b94265ccffa51fc05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:48.833376 9930 consensus_queue.cc:1035] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:48.833936 9930 consensus_queue.cc:1035] T 3c45225f7cfa4821a6f75abc2dd0d43d P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:48.839525 8987 raft_consensus.cc:1261] T 16eeb3fe7ac34432b9dce18727289e1c P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Refusing update from remote peer 4397244002fd4e0f9246b55727b536fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:48.839941 9251 raft_consensus.cc:1261] T 16eeb3fe7ac34432b9dce18727289e1c P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Refusing update from remote peer 4397244002fd4e0f9246b55727b536fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:48.840646 9553 consensus_queue.cc:1035] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:48.841737 9553 consensus_queue.cc:1035] T 16eeb3fe7ac34432b9dce18727289e1c P 4397244002fd4e0f9246b55727b536fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:48.853051 10050 mvcc.cc:205] Tried to move back new op lower bound from 7023240760654893056 to 7023240759138144256. Current Snapshot: MvccSnapshot[applied={T|T < 7023240760654893056}]
I20240502 13:59:48.859030 10043 mvcc.cc:205] Tried to move back new op lower bound from 7023240760654893056 to 7023240759138144256. Current Snapshot: MvccSnapshot[applied={T|T < 7023240760654893056}]
INSERT report
rows total: 2000
time total: 118.574 ms
time per row: 0.059287 ms
I20240502 13:59:48.897799 8574 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54006:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:48.902921 9230 tablet_service.cc:1467] Processing CreateTablet for tablet 6155c2fe97bd41b59314d94e4c3f2a44 (DEFAULT_TABLE table=rebalance_test_table_8 [id=344924bf11894ce0b08b6f8b650f9670]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:48.903239 9230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6155c2fe97bd41b59314d94e4c3f2a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.903633 9099 tablet_service.cc:1467] Processing CreateTablet for tablet 35debb019a2940dfb72d52fcb8c2ddbc (DEFAULT_TABLE table=rebalance_test_table_8 [id=344924bf11894ce0b08b6f8b650f9670]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:48.903801 8702 tablet_service.cc:1467] Processing CreateTablet for tablet 9f39cfbee042492cb621d900a7fdbc8b (DEFAULT_TABLE table=rebalance_test_table_8 [id=344924bf11894ce0b08b6f8b650f9670]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:48.903868 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35debb019a2940dfb72d52fcb8c2ddbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.904002 8702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f39cfbee042492cb621d900a7fdbc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.905459 10071 tablet_bootstrap.cc:492] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360: Bootstrap starting.
I20240502 13:59:48.905683 8965 tablet_service.cc:1467] Processing CreateTablet for tablet 6155c2fe97bd41b59314d94e4c3f2a44 (DEFAULT_TABLE table=rebalance_test_table_8 [id=344924bf11894ce0b08b6f8b650f9670]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:48.905952 8965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6155c2fe97bd41b59314d94e4c3f2a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.906031 10073 tablet_bootstrap.cc:492] T 35debb019a2940dfb72d52fcb8c2ddbc P 2695a61796d043aa823fe3ee966a7541: Bootstrap starting.
I20240502 13:59:48.906219 8703 tablet_service.cc:1467] Processing CreateTablet for tablet 6155c2fe97bd41b59314d94e4c3f2a44 (DEFAULT_TABLE table=rebalance_test_table_8 [id=344924bf11894ce0b08b6f8b650f9670]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:48.906476 10071 tablet_bootstrap.cc:654] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.906924 8967 tablet_service.cc:1467] Processing CreateTablet for tablet 9f39cfbee042492cb621d900a7fdbc8b (DEFAULT_TABLE table=rebalance_test_table_8 [id=344924bf11894ce0b08b6f8b650f9670]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:48.907124 10073 tablet_bootstrap.cc:654] T 35debb019a2940dfb72d52fcb8c2ddbc P 2695a61796d043aa823fe3ee966a7541: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.907168 8967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f39cfbee042492cb621d900a7fdbc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.907441 10071 tablet_bootstrap.cc:492] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360: No bootstrap required, opened a new log
I20240502 13:59:48.907519 10071 ts_tablet_manager.cc:1367] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:48.907864 10071 raft_consensus.cc:348] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.907943 10071 raft_consensus.cc:374] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.907977 10071 raft_consensus.cc:729] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.908099 10071 consensus_queue.cc:260] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.908154 10076 tablet_bootstrap.cc:492] T 6155c2fe97bd41b59314d94e4c3f2a44 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Bootstrap starting.
I20240502 13:59:48.908375 10071 ts_tablet_manager.cc:1398] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:48.908331 8834 tablet_service.cc:1467] Processing CreateTablet for tablet 35debb019a2940dfb72d52fcb8c2ddbc (DEFAULT_TABLE table=rebalance_test_table_8 [id=344924bf11894ce0b08b6f8b650f9670]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:48.908591 8834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35debb019a2940dfb72d52fcb8c2ddbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.908965 10076 tablet_bootstrap.cc:654] T 6155c2fe97bd41b59314d94e4c3f2a44 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.909562 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6155c2fe97bd41b59314d94e4c3f2a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.909775 10076 tablet_bootstrap.cc:492] T 6155c2fe97bd41b59314d94e4c3f2a44 P 66ef9d15c0bd46fbb23d5ba11a085ba6: No bootstrap required, opened a new log
I20240502 13:59:48.909845 10076 ts_tablet_manager.cc:1367] T 6155c2fe97bd41b59314d94e4c3f2a44 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:48.910285 10076 raft_consensus.cc:348] T 6155c2fe97bd41b59314d94e4c3f2a44 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.910439 10076 raft_consensus.cc:374] T 6155c2fe97bd41b59314d94e4c3f2a44 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.910476 10076 raft_consensus.cc:729] T 6155c2fe97bd41b59314d94e4c3f2a44 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.910591 10076 consensus_queue.cc:260] T 6155c2fe97bd41b59314d94e4c3f2a44 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.910848 10076 ts_tablet_manager.cc:1398] T 6155c2fe97bd41b59314d94e4c3f2a44 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:48.911016 10076 tablet_bootstrap.cc:492] T 9f39cfbee042492cb621d900a7fdbc8b P 66ef9d15c0bd46fbb23d5ba11a085ba6: Bootstrap starting.
I20240502 13:59:48.911367 9363 tablet_service.cc:1467] Processing CreateTablet for tablet 35debb019a2940dfb72d52fcb8c2ddbc (DEFAULT_TABLE table=rebalance_test_table_8 [id=344924bf11894ce0b08b6f8b650f9670]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:48.911617 9363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35debb019a2940dfb72d52fcb8c2ddbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.911780 10076 tablet_bootstrap.cc:654] T 9f39cfbee042492cb621d900a7fdbc8b P 66ef9d15c0bd46fbb23d5ba11a085ba6: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.913009 10073 tablet_bootstrap.cc:492] T 35debb019a2940dfb72d52fcb8c2ddbc P 2695a61796d043aa823fe3ee966a7541: No bootstrap required, opened a new log
I20240502 13:59:48.913107 10073 ts_tablet_manager.cc:1367] T 35debb019a2940dfb72d52fcb8c2ddbc P 2695a61796d043aa823fe3ee966a7541: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 13:59:48.913347 10078 tablet_bootstrap.cc:492] T 35debb019a2940dfb72d52fcb8c2ddbc P 379b5e0287244d6b94265ccffa51fc05: Bootstrap starting.
I20240502 13:59:48.913868 10073 raft_consensus.cc:348] T 35debb019a2940dfb72d52fcb8c2ddbc P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:48.913967 10073 raft_consensus.cc:374] T 35debb019a2940dfb72d52fcb8c2ddbc P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.914005 10073 raft_consensus.cc:729] T 35debb019a2940dfb72d52fcb8c2ddbc P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.914125 10073 consensus_queue.cc:260] T 35debb019a2940dfb72d52fcb8c2ddbc P 2695a61796d043aa823fe3ee966a7541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:48.914377 10078 tablet_bootstrap.cc:654] T 35debb019a2940dfb72d52fcb8c2ddbc P 379b5e0287244d6b94265ccffa51fc05: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.914487 10073 ts_tablet_manager.cc:1398] T 35debb019a2940dfb72d52fcb8c2ddbc P 2695a61796d043aa823fe3ee966a7541: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:48.914510 9230 tablet_service.cc:1467] Processing CreateTablet for tablet 9f39cfbee042492cb621d900a7fdbc8b (DEFAULT_TABLE table=rebalance_test_table_8 [id=344924bf11894ce0b08b6f8b650f9670]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:48.914784 9230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f39cfbee042492cb621d900a7fdbc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:48.915483 10072 tablet_bootstrap.cc:492] T 9f39cfbee042492cb621d900a7fdbc8b P 4397244002fd4e0f9246b55727b536fe: Bootstrap starting.
I20240502 13:59:48.916127 10076 tablet_bootstrap.cc:492] T 9f39cfbee042492cb621d900a7fdbc8b P 66ef9d15c0bd46fbb23d5ba11a085ba6: No bootstrap required, opened a new log
I20240502 13:59:48.916209 10076 ts_tablet_manager.cc:1367] T 9f39cfbee042492cb621d900a7fdbc8b P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 13:59:48.916373 10072 tablet_bootstrap.cc:654] T 9f39cfbee042492cb621d900a7fdbc8b P 4397244002fd4e0f9246b55727b536fe: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.916713 10071 tablet_bootstrap.cc:492] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360: Bootstrap starting.
I20240502 13:59:48.916718 10078 tablet_bootstrap.cc:492] T 35debb019a2940dfb72d52fcb8c2ddbc P 379b5e0287244d6b94265ccffa51fc05: No bootstrap required, opened a new log
I20240502 13:59:48.916733 10076 raft_consensus.cc:348] T 9f39cfbee042492cb621d900a7fdbc8b P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.916801 10078 ts_tablet_manager.cc:1367] T 35debb019a2940dfb72d52fcb8c2ddbc P 379b5e0287244d6b94265ccffa51fc05: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 13:59:48.916819 10076 raft_consensus.cc:374] T 9f39cfbee042492cb621d900a7fdbc8b P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.916849 10076 raft_consensus.cc:729] T 9f39cfbee042492cb621d900a7fdbc8b P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.916981 10076 consensus_queue.cc:260] T 9f39cfbee042492cb621d900a7fdbc8b P 66ef9d15c0bd46fbb23d5ba11a085ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.917069 10080 tablet_bootstrap.cc:492] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e: Bootstrap starting.
I20240502 13:59:48.917269 10076 ts_tablet_manager.cc:1398] T 9f39cfbee042492cb621d900a7fdbc8b P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:48.917577 10071 tablet_bootstrap.cc:654] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.917974 10080 tablet_bootstrap.cc:654] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.918015 10078 raft_consensus.cc:348] T 35debb019a2940dfb72d52fcb8c2ddbc P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:48.918121 10078 raft_consensus.cc:374] T 35debb019a2940dfb72d52fcb8c2ddbc P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.918152 10078 raft_consensus.cc:729] T 35debb019a2940dfb72d52fcb8c2ddbc P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.918418 10078 consensus_queue.cc:260] T 35debb019a2940dfb72d52fcb8c2ddbc P 379b5e0287244d6b94265ccffa51fc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:48.919219 10072 tablet_bootstrap.cc:492] T 9f39cfbee042492cb621d900a7fdbc8b P 4397244002fd4e0f9246b55727b536fe: No bootstrap required, opened a new log
I20240502 13:59:48.919283 10078 ts_tablet_manager.cc:1398] T 35debb019a2940dfb72d52fcb8c2ddbc P 379b5e0287244d6b94265ccffa51fc05: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:48.919293 10072 ts_tablet_manager.cc:1367] T 9f39cfbee042492cb621d900a7fdbc8b P 4397244002fd4e0f9246b55727b536fe: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 13:59:48.919759 10072 raft_consensus.cc:348] T 9f39cfbee042492cb621d900a7fdbc8b P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.919845 10072 raft_consensus.cc:374] T 9f39cfbee042492cb621d900a7fdbc8b P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.919865 10072 raft_consensus.cc:729] T 9f39cfbee042492cb621d900a7fdbc8b P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.919970 10072 consensus_queue.cc:260] T 9f39cfbee042492cb621d900a7fdbc8b P 4397244002fd4e0f9246b55727b536fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.920110 10071 tablet_bootstrap.cc:492] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360: No bootstrap required, opened a new log
I20240502 13:59:48.920202 10071 ts_tablet_manager.cc:1367] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 13:59:48.920248 10072 ts_tablet_manager.cc:1398] T 9f39cfbee042492cb621d900a7fdbc8b P 4397244002fd4e0f9246b55727b536fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:48.920403 10072 tablet_bootstrap.cc:492] T 6155c2fe97bd41b59314d94e4c3f2a44 P 4397244002fd4e0f9246b55727b536fe: Bootstrap starting.
I20240502 13:59:48.920585 10071 raft_consensus.cc:348] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.920670 10071 raft_consensus.cc:374] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.920696 10071 raft_consensus.cc:729] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.920789 10071 consensus_queue.cc:260] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.921005 10071 ts_tablet_manager.cc:1398] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:48.921247 10072 tablet_bootstrap.cc:654] T 6155c2fe97bd41b59314d94e4c3f2a44 P 4397244002fd4e0f9246b55727b536fe: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:48.922231 10080 tablet_bootstrap.cc:492] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e: No bootstrap required, opened a new log
I20240502 13:59:48.922386 10080 ts_tablet_manager.cc:1367] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 13:59:48.923086 10072 tablet_bootstrap.cc:492] T 6155c2fe97bd41b59314d94e4c3f2a44 P 4397244002fd4e0f9246b55727b536fe: No bootstrap required, opened a new log
I20240502 13:59:48.923174 10072 ts_tablet_manager.cc:1367] T 6155c2fe97bd41b59314d94e4c3f2a44 P 4397244002fd4e0f9246b55727b536fe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 13:59:48.923287 10080 raft_consensus.cc:348] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:48.923378 10080 raft_consensus.cc:374] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.923398 10080 raft_consensus.cc:729] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.923512 10080 consensus_queue.cc:260] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:48.923591 10072 raft_consensus.cc:348] T 6155c2fe97bd41b59314d94e4c3f2a44 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.923679 10072 raft_consensus.cc:374] T 6155c2fe97bd41b59314d94e4c3f2a44 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:48.923707 10072 raft_consensus.cc:729] T 6155c2fe97bd41b59314d94e4c3f2a44 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Initialized, Role: FOLLOWER
I20240502 13:59:48.923764 10080 ts_tablet_manager.cc:1398] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:48.923816 10072 consensus_queue.cc:260] T 6155c2fe97bd41b59314d94e4c3f2a44 P 4397244002fd4e0f9246b55727b536fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:48.924046 10072 ts_tablet_manager.cc:1398] T 6155c2fe97bd41b59314d94e4c3f2a44 P 4397244002fd4e0f9246b55727b536fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:49.065716 9722 raft_consensus.cc:482] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:49.065866 9722 raft_consensus.cc:504] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:49.066303 9722 leader_election.cc:288] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873), 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573)
I20240502 13:59:49.066704 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6155c2fe97bd41b59314d94e4c3f2a44" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe" is_pre_election: true
I20240502 13:59:49.066706 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6155c2fe97bd41b59314d94e4c3f2a44" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" is_pre_election: true
I20240502 13:59:49.066841 8723 raft_consensus.cc:2454] T 6155c2fe97bd41b59314d94e4c3f2a44 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 0.
I20240502 13:59:49.066857 8987 raft_consensus.cc:2454] T 6155c2fe97bd41b59314d94e4c3f2a44 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 0.
I20240502 13:59:49.067150 9186 leader_election.cc:302] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66ef9d15c0bd46fbb23d5ba11a085ba6, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:49.067353 9722 raft_consensus.cc:2781] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:49.067433 9722 raft_consensus.cc:482] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:49.067482 9722 raft_consensus.cc:3037] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:49.068504 9722 raft_consensus.cc:504] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:49.068826 9722 leader_election.cc:288] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 election: Requested vote from peers 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873), 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573)
I20240502 13:59:49.069188 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6155c2fe97bd41b59314d94e4c3f2a44" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe"
I20240502 13:59:49.069233 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6155c2fe97bd41b59314d94e4c3f2a44" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6"
I20240502 13:59:49.069291 8723 raft_consensus.cc:3037] T 6155c2fe97bd41b59314d94e4c3f2a44 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:49.069355 8987 raft_consensus.cc:3037] T 6155c2fe97bd41b59314d94e4c3f2a44 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:49.070006 8723 raft_consensus.cc:2454] T 6155c2fe97bd41b59314d94e4c3f2a44 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 1.
I20240502 13:59:49.070102 8987 raft_consensus.cc:2454] T 6155c2fe97bd41b59314d94e4c3f2a44 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 1.
I20240502 13:59:49.070259 9185 leader_election.cc:302] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:49.070516 9722 raft_consensus.cc:2781] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:49.070693 9722 raft_consensus.cc:686] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 LEADER]: Becoming Leader. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Running, Role: LEADER
I20240502 13:59:49.070829 9722 consensus_queue.cc:237] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:49.072499 8581 catalog_manager.cc:5484] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133). New cstate: current_term: 1 leader_uuid: "b0a54f7311df4dd2be068d3b15a4a360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:49.147976 9988 raft_consensus.cc:482] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:49.148114 9988 raft_consensus.cc:504] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:49.148499 9988 leader_election.cc:288] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939)
I20240502 13:59:49.148880 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35debb019a2940dfb72d52fcb8c2ddbc" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541" is_pre_election: true
I20240502 13:59:49.148914 8855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35debb019a2940dfb72d52fcb8c2ddbc" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05" is_pre_election: true
I20240502 13:59:49.149024 9119 raft_consensus.cc:2454] T 35debb019a2940dfb72d52fcb8c2ddbc P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 0.
I20240502 13:59:49.149029 8855 raft_consensus.cc:2454] T 35debb019a2940dfb72d52fcb8c2ddbc P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 0.
I20240502 13:59:49.149282 9317 leader_election.cc:302] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:49.149420 9988 raft_consensus.cc:2781] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:49.149477 9988 raft_consensus.cc:482] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:49.149502 9988 raft_consensus.cc:3037] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:49.150242 9988 raft_consensus.cc:504] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:49.150524 9988 leader_election.cc:288] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 election: Requested vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939)
I20240502 13:59:49.150820 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35debb019a2940dfb72d52fcb8c2ddbc" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541"
I20240502 13:59:49.150856 8855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35debb019a2940dfb72d52fcb8c2ddbc" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05"
I20240502 13:59:49.150931 9119 raft_consensus.cc:3037] T 35debb019a2940dfb72d52fcb8c2ddbc P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:49.150947 8855 raft_consensus.cc:3037] T 35debb019a2940dfb72d52fcb8c2ddbc P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:49.151844 8855 raft_consensus.cc:2454] T 35debb019a2940dfb72d52fcb8c2ddbc P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 1.
I20240502 13:59:49.151844 9119 raft_consensus.cc:2454] T 35debb019a2940dfb72d52fcb8c2ddbc P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 1.
I20240502 13:59:49.152123 9317 leader_election.cc:302] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 379b5e0287244d6b94265ccffa51fc05, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:49.152267 9988 raft_consensus.cc:2781] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:49.152503 9988 raft_consensus.cc:686] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 LEADER]: Becoming Leader. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Running, Role: LEADER
I20240502 13:59:49.152643 9988 consensus_queue.cc:237] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:49.154348 8581 catalog_manager.cc:5484] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134). New cstate: current_term: 1 leader_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:49.208442 9722 raft_consensus.cc:482] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:49.208562 9722 raft_consensus.cc:504] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:49.208926 9722 leader_election.cc:288] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873), 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573)
I20240502 13:59:49.209288 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f39cfbee042492cb621d900a7fdbc8b" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" is_pre_election: true
I20240502 13:59:49.209316 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f39cfbee042492cb621d900a7fdbc8b" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe" is_pre_election: true
I20240502 13:59:49.209436 8987 raft_consensus.cc:2454] T 9f39cfbee042492cb621d900a7fdbc8b P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 0.
I20240502 13:59:49.209455 8723 raft_consensus.cc:2454] T 9f39cfbee042492cb621d900a7fdbc8b P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 0.
I20240502 13:59:49.209729 9185 leader_election.cc:302] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:49.209910 9722 raft_consensus.cc:2781] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:49.209978 9722 raft_consensus.cc:482] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:49.210021 9722 raft_consensus.cc:3037] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:49.210850 9722 raft_consensus.cc:504] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:49.211160 9722 leader_election.cc:288] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 election: Requested vote from peers 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873), 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573)
I20240502 13:59:49.211489 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f39cfbee042492cb621d900a7fdbc8b" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe"
I20240502 13:59:49.211508 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f39cfbee042492cb621d900a7fdbc8b" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6"
I20240502 13:59:49.211609 8723 raft_consensus.cc:3037] T 9f39cfbee042492cb621d900a7fdbc8b P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:49.211609 8987 raft_consensus.cc:3037] T 9f39cfbee042492cb621d900a7fdbc8b P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:49.212549 8723 raft_consensus.cc:2454] T 9f39cfbee042492cb621d900a7fdbc8b P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 1.
I20240502 13:59:49.212549 8987 raft_consensus.cc:2454] T 9f39cfbee042492cb621d900a7fdbc8b P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 1.
I20240502 13:59:49.212908 9185 leader_election.cc:302] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:49.213049 9722 raft_consensus.cc:2781] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:49.213150 9722 raft_consensus.cc:686] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 [term 1 LEADER]: Becoming Leader. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Running, Role: LEADER
I20240502 13:59:49.213336 9722 consensus_queue.cc:237] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:49.215137 8581 catalog_manager.cc:5484] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133). New cstate: current_term: 1 leader_uuid: "b0a54f7311df4dd2be068d3b15a4a360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:49.385092 10091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:49.385356 10091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:49.389407 10091 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 13:59:49.473340 8987 raft_consensus.cc:1261] T 9f39cfbee042492cb621d900a7fdbc8b P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Refusing update from remote peer b0a54f7311df4dd2be068d3b15a4a360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:49.473332 8723 raft_consensus.cc:1261] T 9f39cfbee042492cb621d900a7fdbc8b P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Refusing update from remote peer b0a54f7311df4dd2be068d3b15a4a360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:49.473766 10086 consensus_queue.cc:1035] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:49.474073 9722 consensus_queue.cc:1035] T 9f39cfbee042492cb621d900a7fdbc8b P b0a54f7311df4dd2be068d3b15a4a360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:49.474627 9119 raft_consensus.cc:1261] T 35debb019a2940dfb72d52fcb8c2ddbc P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Refusing update from remote peer 833c2ad4636d4d8e9861339a4ad9e80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:49.474627 8855 raft_consensus.cc:1261] T 35debb019a2940dfb72d52fcb8c2ddbc P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Refusing update from remote peer 833c2ad4636d4d8e9861339a4ad9e80e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:49.475091 10088 consensus_queue.cc:1035] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:49.476358 10088 consensus_queue.cc:1035] T 35debb019a2940dfb72d52fcb8c2ddbc P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:49.480072 8986 raft_consensus.cc:1261] T 6155c2fe97bd41b59314d94e4c3f2a44 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Refusing update from remote peer b0a54f7311df4dd2be068d3b15a4a360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:49.480423 8722 raft_consensus.cc:1261] T 6155c2fe97bd41b59314d94e4c3f2a44 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Refusing update from remote peer b0a54f7311df4dd2be068d3b15a4a360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:49.480912 9722 consensus_queue.cc:1035] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:49.481196 10086 consensus_queue.cc:1035] T 6155c2fe97bd41b59314d94e4c3f2a44 P b0a54f7311df4dd2be068d3b15a4a360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:49.488876 10125 mvcc.cc:205] Tried to move back new op lower bound from 7023240763278872576 to 7023240762219405312. Current Snapshot: MvccSnapshot[applied={T|T < 7023240763278872576}]
I20240502 13:59:49.496831 10123 mvcc.cc:205] Tried to move back new op lower bound from 7023240763335536640 to 7023240762219405312. Current Snapshot: MvccSnapshot[applied={T|T < 7023240763278872576}]
I20240502 13:59:49.498903 10131 mvcc.cc:205] Tried to move back new op lower bound from 7023240763283673088 to 7023240761970544640. Current Snapshot: MvccSnapshot[applied={T|T < 7023240763283673088}]
INSERT report
rows total: 2000
time total: 128.822 ms
time per row: 0.064411 ms
I20240502 13:59:49.556972 8581 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54006:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:49.561947 8834 tablet_service.cc:1467] Processing CreateTablet for tablet badff6c7db104260924db424787cac2a (DEFAULT_TABLE table=rebalance_test_table_9 [id=0bbbb130189147428d9bb5b7c992d593]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:49.562310 8834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet badff6c7db104260924db424787cac2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:49.562265 9099 tablet_service.cc:1467] Processing CreateTablet for tablet badff6c7db104260924db424787cac2a (DEFAULT_TABLE table=rebalance_test_table_9 [id=0bbbb130189147428d9bb5b7c992d593]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:49.562558 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet badff6c7db104260924db424787cac2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:49.562567 9362 tablet_service.cc:1467] Processing CreateTablet for tablet 4368b81254a24d1cba21912300deb7db (DEFAULT_TABLE table=rebalance_test_table_9 [id=0bbbb130189147428d9bb5b7c992d593]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:49.562804 9362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4368b81254a24d1cba21912300deb7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:49.564574 10147 tablet_bootstrap.cc:492] T badff6c7db104260924db424787cac2a P 379b5e0287244d6b94265ccffa51fc05: Bootstrap starting.
I20240502 13:59:49.564714 9363 tablet_service.cc:1467] Processing CreateTablet for tablet badff6c7db104260924db424787cac2a (DEFAULT_TABLE table=rebalance_test_table_9 [id=0bbbb130189147428d9bb5b7c992d593]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:49.564944 9363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet badff6c7db104260924db424787cac2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:49.565091 10149 tablet_bootstrap.cc:492] T 4368b81254a24d1cba21912300deb7db P 833c2ad4636d4d8e9861339a4ad9e80e: Bootstrap starting.
I20240502 13:59:49.565483 9230 tablet_service.cc:1467] Processing CreateTablet for tablet 66671d0baed54d17bc6f3ce1535d8550 (DEFAULT_TABLE table=rebalance_test_table_9 [id=0bbbb130189147428d9bb5b7c992d593]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:49.565572 10147 tablet_bootstrap.cc:654] T badff6c7db104260924db424787cac2a P 379b5e0287244d6b94265ccffa51fc05: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:49.565622 8967 tablet_service.cc:1467] Processing CreateTablet for tablet 66671d0baed54d17bc6f3ce1535d8550 (DEFAULT_TABLE table=rebalance_test_table_9 [id=0bbbb130189147428d9bb5b7c992d593]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:49.565769 9230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66671d0baed54d17bc6f3ce1535d8550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:49.565879 8967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66671d0baed54d17bc6f3ce1535d8550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:49.565999 10149 tablet_bootstrap.cc:654] T 4368b81254a24d1cba21912300deb7db P 833c2ad4636d4d8e9861339a4ad9e80e: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:49.567049 10149 tablet_bootstrap.cc:492] T 4368b81254a24d1cba21912300deb7db P 833c2ad4636d4d8e9861339a4ad9e80e: No bootstrap required, opened a new log
I20240502 13:59:49.567140 10149 ts_tablet_manager.cc:1367] T 4368b81254a24d1cba21912300deb7db P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:49.567574 10149 raft_consensus.cc:348] T 4368b81254a24d1cba21912300deb7db P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:49.567708 10149 raft_consensus.cc:374] T 4368b81254a24d1cba21912300deb7db P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:49.567744 10149 raft_consensus.cc:729] T 4368b81254a24d1cba21912300deb7db P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Initialized, Role: FOLLOWER
I20240502 13:59:49.567847 10149 consensus_queue.cc:260] T 4368b81254a24d1cba21912300deb7db P 833c2ad4636d4d8e9861339a4ad9e80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:49.568125 10149 ts_tablet_manager.cc:1398] T 4368b81254a24d1cba21912300deb7db P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:49.568270 10149 tablet_bootstrap.cc:492] T badff6c7db104260924db424787cac2a P 833c2ad4636d4d8e9861339a4ad9e80e: Bootstrap starting.
I20240502 13:59:49.568254 10152 tablet_bootstrap.cc:492] T 66671d0baed54d17bc6f3ce1535d8550 P b0a54f7311df4dd2be068d3b15a4a360: Bootstrap starting.
I20240502 13:59:49.568935 10147 tablet_bootstrap.cc:492] T badff6c7db104260924db424787cac2a P 379b5e0287244d6b94265ccffa51fc05: No bootstrap required, opened a new log
I20240502 13:59:49.569018 10147 ts_tablet_manager.cc:1367] T badff6c7db104260924db424787cac2a P 379b5e0287244d6b94265ccffa51fc05: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 13:59:49.569054 10149 tablet_bootstrap.cc:654] T badff6c7db104260924db424787cac2a P 833c2ad4636d4d8e9861339a4ad9e80e: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:49.569212 9099 tablet_service.cc:1467] Processing CreateTablet for tablet 4368b81254a24d1cba21912300deb7db (DEFAULT_TABLE table=rebalance_test_table_9 [id=0bbbb130189147428d9bb5b7c992d593]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:49.569482 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4368b81254a24d1cba21912300deb7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:49.569406 10147 raft_consensus.cc:348] T badff6c7db104260924db424787cac2a P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } }
I20240502 13:59:49.569518 10147 raft_consensus.cc:374] T badff6c7db104260924db424787cac2a P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:49.569546 10147 raft_consensus.cc:729] T badff6c7db104260924db424787cac2a P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Initialized, Role: FOLLOWER
I20240502 13:59:49.569557 10152 tablet_bootstrap.cc:654] T 66671d0baed54d17bc6f3ce1535d8550 P b0a54f7311df4dd2be068d3b15a4a360: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:49.569655 10147 consensus_queue.cc:260] T badff6c7db104260924db424787cac2a P 379b5e0287244d6b94265ccffa51fc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } }
I20240502 13:59:49.569907 10147 ts_tablet_manager.cc:1398] T badff6c7db104260924db424787cac2a P 379b5e0287244d6b94265ccffa51fc05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:49.570192 8834 tablet_service.cc:1467] Processing CreateTablet for tablet 4368b81254a24d1cba21912300deb7db (DEFAULT_TABLE table=rebalance_test_table_9 [id=0bbbb130189147428d9bb5b7c992d593]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:49.570427 8834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4368b81254a24d1cba21912300deb7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:49.570412 8703 tablet_service.cc:1467] Processing CreateTablet for tablet 66671d0baed54d17bc6f3ce1535d8550 (DEFAULT_TABLE table=rebalance_test_table_9 [id=0bbbb130189147428d9bb5b7c992d593]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:49.570652 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66671d0baed54d17bc6f3ce1535d8550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:49.570788 10152 tablet_bootstrap.cc:492] T 66671d0baed54d17bc6f3ce1535d8550 P b0a54f7311df4dd2be068d3b15a4a360: No bootstrap required, opened a new log
I20240502 13:59:49.570853 10152 ts_tablet_manager.cc:1367] T 66671d0baed54d17bc6f3ce1535d8550 P b0a54f7311df4dd2be068d3b15a4a360: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 13:59:49.571205 10152 raft_consensus.cc:348] T 66671d0baed54d17bc6f3ce1535d8550 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:49.571311 10152 raft_consensus.cc:374] T 66671d0baed54d17bc6f3ce1535d8550 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:49.571339 10152 raft_consensus.cc:729] T 66671d0baed54d17bc6f3ce1535d8550 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Initialized, Role: FOLLOWER
I20240502 13:59:49.571434 10152 consensus_queue.cc:260] T 66671d0baed54d17bc6f3ce1535d8550 P b0a54f7311df4dd2be068d3b15a4a360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:49.571732 10152 ts_tablet_manager.cc:1398] T 66671d0baed54d17bc6f3ce1535d8550 P b0a54f7311df4dd2be068d3b15a4a360: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:49.573614 10153 tablet_bootstrap.cc:492] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Bootstrap starting.
I20240502 13:59:49.573947 10147 tablet_bootstrap.cc:492] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05: Bootstrap starting.
I20240502 13:59:49.574532 10149 tablet_bootstrap.cc:492] T badff6c7db104260924db424787cac2a P 833c2ad4636d4d8e9861339a4ad9e80e: No bootstrap required, opened a new log
I20240502 13:59:49.574604 10149 ts_tablet_manager.cc:1367] T badff6c7db104260924db424787cac2a P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 13:59:49.574728 10147 tablet_bootstrap.cc:654] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:49.574887 10153 tablet_bootstrap.cc:654] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:49.574975 10149 raft_consensus.cc:348] T badff6c7db104260924db424787cac2a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } }
I20240502 13:59:49.575067 10149 raft_consensus.cc:374] T badff6c7db104260924db424787cac2a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:49.575094 10149 raft_consensus.cc:729] T badff6c7db104260924db424787cac2a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Initialized, Role: FOLLOWER
I20240502 13:59:49.575203 10149 consensus_queue.cc:260] T badff6c7db104260924db424787cac2a P 833c2ad4636d4d8e9861339a4ad9e80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } }
I20240502 13:59:49.575978 10149 ts_tablet_manager.cc:1398] T badff6c7db104260924db424787cac2a P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:49.576157 10148 tablet_bootstrap.cc:492] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541: Bootstrap starting.
I20240502 13:59:49.576505 10153 tablet_bootstrap.cc:492] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6: No bootstrap required, opened a new log
I20240502 13:59:49.576592 10153 ts_tablet_manager.cc:1367] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 13:59:49.576857 10156 tablet_bootstrap.cc:492] T 66671d0baed54d17bc6f3ce1535d8550 P 4397244002fd4e0f9246b55727b536fe: Bootstrap starting.
I20240502 13:59:49.577008 10153 raft_consensus.cc:348] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:49.577095 10153 raft_consensus.cc:374] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:49.577152 10153 raft_consensus.cc:729] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Initialized, Role: FOLLOWER
I20240502 13:59:49.577301 10153 consensus_queue.cc:260] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:49.577697 10153 ts_tablet_manager.cc:1398] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:49.579012 10156 tablet_bootstrap.cc:654] T 66671d0baed54d17bc6f3ce1535d8550 P 4397244002fd4e0f9246b55727b536fe: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:49.579035 10148 tablet_bootstrap.cc:654] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:49.580183 10156 tablet_bootstrap.cc:492] T 66671d0baed54d17bc6f3ce1535d8550 P 4397244002fd4e0f9246b55727b536fe: No bootstrap required, opened a new log
I20240502 13:59:49.580317 10156 ts_tablet_manager.cc:1367] T 66671d0baed54d17bc6f3ce1535d8550 P 4397244002fd4e0f9246b55727b536fe: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 13:59:49.580713 10156 raft_consensus.cc:348] T 66671d0baed54d17bc6f3ce1535d8550 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:49.580803 10156 raft_consensus.cc:374] T 66671d0baed54d17bc6f3ce1535d8550 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:49.580832 10156 raft_consensus.cc:729] T 66671d0baed54d17bc6f3ce1535d8550 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Initialized, Role: FOLLOWER
I20240502 13:59:49.580938 10156 consensus_queue.cc:260] T 66671d0baed54d17bc6f3ce1535d8550 P 4397244002fd4e0f9246b55727b536fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:49.581220 10156 ts_tablet_manager.cc:1398] T 66671d0baed54d17bc6f3ce1535d8550 P 4397244002fd4e0f9246b55727b536fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:49.590656 10148 tablet_bootstrap.cc:492] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541: No bootstrap required, opened a new log
I20240502 13:59:49.590746 10148 ts_tablet_manager.cc:1367] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240502 13:59:49.591163 10148 raft_consensus.cc:348] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } }
I20240502 13:59:49.591255 10148 raft_consensus.cc:374] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:49.591305 10148 raft_consensus.cc:729] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Initialized, Role: FOLLOWER
I20240502 13:59:49.591434 10148 consensus_queue.cc:260] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } }
I20240502 13:59:49.591670 10148 ts_tablet_manager.cc:1398] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:49.591822 10148 tablet_bootstrap.cc:492] T 4368b81254a24d1cba21912300deb7db P 2695a61796d043aa823fe3ee966a7541: Bootstrap starting.
I20240502 13:59:49.592641 10148 tablet_bootstrap.cc:654] T 4368b81254a24d1cba21912300deb7db P 2695a61796d043aa823fe3ee966a7541: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:49.593659 10148 tablet_bootstrap.cc:492] T 4368b81254a24d1cba21912300deb7db P 2695a61796d043aa823fe3ee966a7541: No bootstrap required, opened a new log
I20240502 13:59:49.593734 10148 ts_tablet_manager.cc:1367] T 4368b81254a24d1cba21912300deb7db P 2695a61796d043aa823fe3ee966a7541: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:49.594069 10148 raft_consensus.cc:348] T 4368b81254a24d1cba21912300deb7db P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:49.594151 10148 raft_consensus.cc:374] T 4368b81254a24d1cba21912300deb7db P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:49.594183 10148 raft_consensus.cc:729] T 4368b81254a24d1cba21912300deb7db P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Initialized, Role: FOLLOWER
I20240502 13:59:49.594398 10148 consensus_queue.cc:260] T 4368b81254a24d1cba21912300deb7db P 2695a61796d043aa823fe3ee966a7541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:49.594606 10147 tablet_bootstrap.cc:492] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05: No bootstrap required, opened a new log
I20240502 13:59:49.594638 10148 ts_tablet_manager.cc:1398] T 4368b81254a24d1cba21912300deb7db P 2695a61796d043aa823fe3ee966a7541: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:49.594695 10147 ts_tablet_manager.cc:1367] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.001s
I20240502 13:59:49.595074 10147 raft_consensus.cc:348] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:49.595182 10147 raft_consensus.cc:374] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:49.595224 10147 raft_consensus.cc:729] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Initialized, Role: FOLLOWER
I20240502 13:59:49.595345 10147 consensus_queue.cc:260] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:49.595609 10147 ts_tablet_manager.cc:1398] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:49.635741 9793 raft_consensus.cc:482] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:49.635856 9793 raft_consensus.cc:504] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } }
I20240502 13:59:49.636255 9793 leader_election.cc:288] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939), 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134:35071)
I20240502 13:59:49.636585 8855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badff6c7db104260924db424787cac2a" candidate_uuid: "2695a61796d043aa823fe3ee966a7541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05" is_pre_election: true
I20240502 13:59:49.636618 9382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badff6c7db104260924db424787cac2a" candidate_uuid: "2695a61796d043aa823fe3ee966a7541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" is_pre_election: true
I20240502 13:59:49.636715 8855 raft_consensus.cc:2454] T badff6c7db104260924db424787cac2a P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2695a61796d043aa823fe3ee966a7541 in term 0.
I20240502 13:59:49.636734 9382 raft_consensus.cc:2454] T badff6c7db104260924db424787cac2a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2695a61796d043aa823fe3ee966a7541 in term 0.
I20240502 13:59:49.637003 9053 leader_election.cc:302] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 379b5e0287244d6b94265ccffa51fc05; no voters:
I20240502 13:59:49.637182 9793 raft_consensus.cc:2781] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:49.637248 9793 raft_consensus.cc:482] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:49.637290 9793 raft_consensus.cc:3037] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:49.638168 9793 raft_consensus.cc:504] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } }
I20240502 13:59:49.638471 9793 leader_election.cc:288] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 [CANDIDATE]: Term 1 election: Requested vote from peers 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939), 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134:35071)
I20240502 13:59:49.638772 8855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badff6c7db104260924db424787cac2a" candidate_uuid: "2695a61796d043aa823fe3ee966a7541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05"
I20240502 13:59:49.638782 9382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "badff6c7db104260924db424787cac2a" candidate_uuid: "2695a61796d043aa823fe3ee966a7541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833c2ad4636d4d8e9861339a4ad9e80e"
I20240502 13:59:49.638865 9382 raft_consensus.cc:3037] T badff6c7db104260924db424787cac2a P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:49.638876 8855 raft_consensus.cc:3037] T badff6c7db104260924db424787cac2a P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:49.639583 9382 raft_consensus.cc:2454] T badff6c7db104260924db424787cac2a P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2695a61796d043aa823fe3ee966a7541 in term 1.
I20240502 13:59:49.639607 8855 raft_consensus.cc:2454] T badff6c7db104260924db424787cac2a P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2695a61796d043aa823fe3ee966a7541 in term 1.
I20240502 13:59:49.639854 9054 leader_election.cc:302] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:49.640003 9793 raft_consensus.cc:2781] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:49.640201 9793 raft_consensus.cc:686] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 [term 1 LEADER]: Becoming Leader. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Running, Role: LEADER
I20240502 13:59:49.640354 9793 consensus_queue.cc:237] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } }
I20240502 13:59:49.642045 8579 catalog_manager.cc:5484] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2695a61796d043aa823fe3ee966a7541 (127.8.86.132). New cstate: current_term: 1 leader_uuid: "2695a61796d043aa823fe3ee966a7541" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:49.653508 10001 raft_consensus.cc:482] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:49.653600 10001 raft_consensus.cc:504] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:49.653913 10001 leader_election.cc:288] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134:35071), 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215)
I20240502 13:59:49.654222 9382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4368b81254a24d1cba21912300deb7db" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" is_pre_election: true
I20240502 13:59:49.654220 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4368b81254a24d1cba21912300deb7db" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541" is_pre_election: true
I20240502 13:59:49.654366 9382 raft_consensus.cc:2454] T 4368b81254a24d1cba21912300deb7db P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 0.
I20240502 13:59:49.654366 9119 raft_consensus.cc:2454] T 4368b81254a24d1cba21912300deb7db P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 0.
I20240502 13:59:49.654615 8790 leader_election.cc:302] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 379b5e0287244d6b94265ccffa51fc05, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:49.654778 10001 raft_consensus.cc:2781] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:49.654839 10001 raft_consensus.cc:482] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:49.654876 10001 raft_consensus.cc:3037] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:49.655583 10001 raft_consensus.cc:504] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:49.655882 10001 leader_election.cc:288] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 election: Requested vote from peers 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134:35071), 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215)
I20240502 13:59:49.656193 9382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4368b81254a24d1cba21912300deb7db" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833c2ad4636d4d8e9861339a4ad9e80e"
I20240502 13:59:49.656193 9119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4368b81254a24d1cba21912300deb7db" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541"
I20240502 13:59:49.656309 9119 raft_consensus.cc:3037] T 4368b81254a24d1cba21912300deb7db P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:49.656309 9382 raft_consensus.cc:3037] T 4368b81254a24d1cba21912300deb7db P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:49.657202 9382 raft_consensus.cc:2454] T 4368b81254a24d1cba21912300deb7db P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 1.
I20240502 13:59:49.657215 9119 raft_consensus.cc:2454] T 4368b81254a24d1cba21912300deb7db P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 1.
I20240502 13:59:49.657459 8790 leader_election.cc:302] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 379b5e0287244d6b94265ccffa51fc05, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:49.657624 10001 raft_consensus.cc:2781] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:49.657827 10001 raft_consensus.cc:686] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 [term 1 LEADER]: Becoming Leader. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Running, Role: LEADER
I20240502 13:59:49.657953 10001 consensus_queue.cc:237] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } }
I20240502 13:59:49.659592 8579 catalog_manager.cc:5484] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "379b5e0287244d6b94265ccffa51fc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:49.744436 9943 raft_consensus.cc:482] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:49.744561 9943 raft_consensus.cc:504] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:49.744983 9943 leader_election.cc:288] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699), 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573)
I20240502 13:59:49.745416 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66671d0baed54d17bc6f3ce1535d8550" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360" is_pre_election: true
I20240502 13:59:49.745419 8722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66671d0baed54d17bc6f3ce1535d8550" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe" is_pre_election: true
I20240502 13:59:49.745580 8722 raft_consensus.cc:2454] T 66671d0baed54d17bc6f3ce1535d8550 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 0.
I20240502 13:59:49.745621 9251 raft_consensus.cc:2454] T 66671d0baed54d17bc6f3ce1535d8550 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 0.
I20240502 13:59:49.745908 8921 leader_election.cc:302] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, 66ef9d15c0bd46fbb23d5ba11a085ba6; no voters:
I20240502 13:59:49.746109 9943 raft_consensus.cc:2781] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:49.746173 9943 raft_consensus.cc:482] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:49.746201 9943 raft_consensus.cc:3037] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:49.747165 9943 raft_consensus.cc:504] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:49.747474 9943 leader_election.cc:288] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 election: Requested vote from peers b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699), 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573)
I20240502 13:59:49.747795 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66671d0baed54d17bc6f3ce1535d8550" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360"
I20240502 13:59:49.747853 8722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66671d0baed54d17bc6f3ce1535d8550" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe"
I20240502 13:59:49.747903 9251 raft_consensus.cc:3037] T 66671d0baed54d17bc6f3ce1535d8550 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:49.747960 8722 raft_consensus.cc:3037] T 66671d0baed54d17bc6f3ce1535d8550 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:49.748757 9251 raft_consensus.cc:2454] T 66671d0baed54d17bc6f3ce1535d8550 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 1.
I20240502 13:59:49.748765 8722 raft_consensus.cc:2454] T 66671d0baed54d17bc6f3ce1535d8550 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 1.
I20240502 13:59:49.749025 8921 leader_election.cc:302] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, 66ef9d15c0bd46fbb23d5ba11a085ba6; no voters:
I20240502 13:59:49.749192 9943 raft_consensus.cc:2781] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:49.749406 9943 raft_consensus.cc:686] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 LEADER]: Becoming Leader. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Running, Role: LEADER
I20240502 13:59:49.749575 9943 consensus_queue.cc:237] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:49.751451 8579 catalog_manager.cc:5484] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131). New cstate: current_term: 1 leader_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:49.874102 10168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:49.874361 10168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:49.877382 10168 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 13:59:49.967959 9382 raft_consensus.cc:1261] T badff6c7db104260924db424787cac2a P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Refusing update from remote peer 2695a61796d043aa823fe3ee966a7541: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:49.968452 9793 consensus_queue.cc:1035] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:49.969439 8855 raft_consensus.cc:1261] T badff6c7db104260924db424787cac2a P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Refusing update from remote peer 2695a61796d043aa823fe3ee966a7541: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:49.973248 8722 raft_consensus.cc:1261] T 66671d0baed54d17bc6f3ce1535d8550 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Refusing update from remote peer 66ef9d15c0bd46fbb23d5ba11a085ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:49.973351 9251 raft_consensus.cc:1261] T 66671d0baed54d17bc6f3ce1535d8550 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Refusing update from remote peer 66ef9d15c0bd46fbb23d5ba11a085ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:49.974021 9383 raft_consensus.cc:1261] T 4368b81254a24d1cba21912300deb7db P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Refusing update from remote peer 379b5e0287244d6b94265ccffa51fc05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:49.974270 10166 consensus_queue.cc:1035] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:49.974637 10166 consensus_queue.cc:1035] T 66671d0baed54d17bc6f3ce1535d8550 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:49.975189 9118 raft_consensus.cc:1261] T 4368b81254a24d1cba21912300deb7db P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Refusing update from remote peer 379b5e0287244d6b94265ccffa51fc05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:49.975539 10001 consensus_queue.cc:1035] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:49.975921 10164 consensus_queue.cc:1035] T 4368b81254a24d1cba21912300deb7db P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:49.982424 10162 consensus_queue.cc:1035] T badff6c7db104260924db424787cac2a P 2695a61796d043aa823fe3ee966a7541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 134.203 ms
time per row: 0.0671014 ms
I20240502 13:59:50.057790 8572 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54006:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:50.063405 9230 tablet_service.cc:1467] Processing CreateTablet for tablet 96319242e7d6416bbe63cce24c90e30f (DEFAULT_TABLE table=rebalance_test_table_10 [id=f08097e195064822b99ebb50ab91a9be]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:50.063526 8967 tablet_service.cc:1467] Processing CreateTablet for tablet 96319242e7d6416bbe63cce24c90e30f (DEFAULT_TABLE table=rebalance_test_table_10 [id=f08097e195064822b99ebb50ab91a9be]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:50.063742 9230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96319242e7d6416bbe63cce24c90e30f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.063813 8967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96319242e7d6416bbe63cce24c90e30f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.064396 8703 tablet_service.cc:1467] Processing CreateTablet for tablet 96319242e7d6416bbe63cce24c90e30f (DEFAULT_TABLE table=rebalance_test_table_10 [id=f08097e195064822b99ebb50ab91a9be]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:50.064662 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96319242e7d6416bbe63cce24c90e30f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.064597 8834 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab480ba1dd34ca5b6694125374182dd (DEFAULT_TABLE table=rebalance_test_table_10 [id=f08097e195064822b99ebb50ab91a9be]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:50.064888 8834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab480ba1dd34ca5b6694125374182dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.066020 10152 tablet_bootstrap.cc:492] T 96319242e7d6416bbe63cce24c90e30f P b0a54f7311df4dd2be068d3b15a4a360: Bootstrap starting.
I20240502 13:59:50.066217 9099 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab480ba1dd34ca5b6694125374182dd (DEFAULT_TABLE table=rebalance_test_table_10 [id=f08097e195064822b99ebb50ab91a9be]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:50.066526 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab480ba1dd34ca5b6694125374182dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.067049 10152 tablet_bootstrap.cc:654] T 96319242e7d6416bbe63cce24c90e30f P b0a54f7311df4dd2be068d3b15a4a360: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:50.067061 10147 tablet_bootstrap.cc:492] T 9ab480ba1dd34ca5b6694125374182dd P 379b5e0287244d6b94265ccffa51fc05: Bootstrap starting.
I20240502 13:59:50.067585 10156 tablet_bootstrap.cc:492] T 96319242e7d6416bbe63cce24c90e30f P 4397244002fd4e0f9246b55727b536fe: Bootstrap starting.
I20240502 13:59:50.068006 10147 tablet_bootstrap.cc:654] T 9ab480ba1dd34ca5b6694125374182dd P 379b5e0287244d6b94265ccffa51fc05: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:50.068418 10148 tablet_bootstrap.cc:492] T 9ab480ba1dd34ca5b6694125374182dd P 2695a61796d043aa823fe3ee966a7541: Bootstrap starting.
I20240502 13:59:50.068511 10156 tablet_bootstrap.cc:654] T 96319242e7d6416bbe63cce24c90e30f P 4397244002fd4e0f9246b55727b536fe: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:50.068656 8702 tablet_service.cc:1467] Processing CreateTablet for tablet 6b48981997f145c581bf21555310d569 (DEFAULT_TABLE table=rebalance_test_table_10 [id=f08097e195064822b99ebb50ab91a9be]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:50.068884 8702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b48981997f145c581bf21555310d569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.069197 10148 tablet_bootstrap.cc:654] T 9ab480ba1dd34ca5b6694125374182dd P 2695a61796d043aa823fe3ee966a7541: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:50.069558 10156 tablet_bootstrap.cc:492] T 96319242e7d6416bbe63cce24c90e30f P 4397244002fd4e0f9246b55727b536fe: No bootstrap required, opened a new log
I20240502 13:59:50.069644 10156 ts_tablet_manager.cc:1367] T 96319242e7d6416bbe63cce24c90e30f P 4397244002fd4e0f9246b55727b536fe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:50.069947 10147 tablet_bootstrap.cc:492] T 9ab480ba1dd34ca5b6694125374182dd P 379b5e0287244d6b94265ccffa51fc05: No bootstrap required, opened a new log
I20240502 13:59:50.070040 10147 ts_tablet_manager.cc:1367] T 9ab480ba1dd34ca5b6694125374182dd P 379b5e0287244d6b94265ccffa51fc05: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 13:59:50.070117 10156 raft_consensus.cc:348] T 96319242e7d6416bbe63cce24c90e30f P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:50.070240 10156 raft_consensus.cc:374] T 96319242e7d6416bbe63cce24c90e30f P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:50.070277 10156 raft_consensus.cc:729] T 96319242e7d6416bbe63cce24c90e30f P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.070441 10156 consensus_queue.cc:260] T 96319242e7d6416bbe63cce24c90e30f P 4397244002fd4e0f9246b55727b536fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:50.070725 10156 ts_tablet_manager.cc:1398] T 96319242e7d6416bbe63cce24c90e30f P 4397244002fd4e0f9246b55727b536fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:50.070691 10147 raft_consensus.cc:348] T 9ab480ba1dd34ca5b6694125374182dd P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.070788 10147 raft_consensus.cc:374] T 9ab480ba1dd34ca5b6694125374182dd P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:50.070811 10148 tablet_bootstrap.cc:492] T 9ab480ba1dd34ca5b6694125374182dd P 2695a61796d043aa823fe3ee966a7541: No bootstrap required, opened a new log
I20240502 13:59:50.070820 10147 raft_consensus.cc:729] T 9ab480ba1dd34ca5b6694125374182dd P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.070891 10148 ts_tablet_manager.cc:1367] T 9ab480ba1dd34ca5b6694125374182dd P 2695a61796d043aa823fe3ee966a7541: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 13:59:50.070947 10147 consensus_queue.cc:260] T 9ab480ba1dd34ca5b6694125374182dd P 379b5e0287244d6b94265ccffa51fc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.071079 10153 tablet_bootstrap.cc:492] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6: Bootstrap starting.
I20240502 13:59:50.071194 10147 ts_tablet_manager.cc:1398] T 9ab480ba1dd34ca5b6694125374182dd P 379b5e0287244d6b94265ccffa51fc05: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:50.071308 10148 raft_consensus.cc:348] T 9ab480ba1dd34ca5b6694125374182dd P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.071400 10148 raft_consensus.cc:374] T 9ab480ba1dd34ca5b6694125374182dd P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:50.071430 10148 raft_consensus.cc:729] T 9ab480ba1dd34ca5b6694125374182dd P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.071563 10148 consensus_queue.cc:260] T 9ab480ba1dd34ca5b6694125374182dd P 2695a61796d043aa823fe3ee966a7541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.071825 10148 ts_tablet_manager.cc:1398] T 9ab480ba1dd34ca5b6694125374182dd P 2695a61796d043aa823fe3ee966a7541: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:50.072010 10153 tablet_bootstrap.cc:654] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:50.073246 9363 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab480ba1dd34ca5b6694125374182dd (DEFAULT_TABLE table=rebalance_test_table_10 [id=f08097e195064822b99ebb50ab91a9be]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:50.073400 9230 tablet_service.cc:1467] Processing CreateTablet for tablet 6b48981997f145c581bf21555310d569 (DEFAULT_TABLE table=rebalance_test_table_10 [id=f08097e195064822b99ebb50ab91a9be]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:50.073534 9363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab480ba1dd34ca5b6694125374182dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.073658 9230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b48981997f145c581bf21555310d569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.073976 8967 tablet_service.cc:1467] Processing CreateTablet for tablet 6b48981997f145c581bf21555310d569 (DEFAULT_TABLE table=rebalance_test_table_10 [id=f08097e195064822b99ebb50ab91a9be]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:50.074211 8967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b48981997f145c581bf21555310d569. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.074474 10156 tablet_bootstrap.cc:492] T 6b48981997f145c581bf21555310d569 P 4397244002fd4e0f9246b55727b536fe: Bootstrap starting.
I20240502 13:59:50.076236 10149 tablet_bootstrap.cc:492] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e: Bootstrap starting.
I20240502 13:59:50.076938 10149 tablet_bootstrap.cc:654] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:50.077405 10153 tablet_bootstrap.cc:492] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6: No bootstrap required, opened a new log
I20240502 13:59:50.077497 10153 ts_tablet_manager.cc:1367] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 13:59:50.077976 10153 raft_consensus.cc:348] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:50.078069 10149 tablet_bootstrap.cc:492] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e: No bootstrap required, opened a new log
I20240502 13:59:50.078083 10153 raft_consensus.cc:374] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:50.078119 10153 raft_consensus.cc:729] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.078157 10149 ts_tablet_manager.cc:1367] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:50.075644 10156 tablet_bootstrap.cc:654] T 6b48981997f145c581bf21555310d569 P 4397244002fd4e0f9246b55727b536fe: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:50.078230 10153 consensus_queue.cc:260] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:50.078541 10153 ts_tablet_manager.cc:1398] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:50.078601 10149 raft_consensus.cc:348] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.078681 10149 raft_consensus.cc:374] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:50.078688 10153 tablet_bootstrap.cc:492] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Bootstrap starting.
I20240502 13:59:50.078701 10149 raft_consensus.cc:729] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.078809 10149 consensus_queue.cc:260] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.079061 10149 ts_tablet_manager.cc:1398] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:50.079566 10153 tablet_bootstrap.cc:654] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:50.079937 10152 tablet_bootstrap.cc:492] T 96319242e7d6416bbe63cce24c90e30f P b0a54f7311df4dd2be068d3b15a4a360: No bootstrap required, opened a new log
I20240502 13:59:50.080027 10152 ts_tablet_manager.cc:1367] T 96319242e7d6416bbe63cce24c90e30f P b0a54f7311df4dd2be068d3b15a4a360: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240502 13:59:50.080458 10152 raft_consensus.cc:348] T 96319242e7d6416bbe63cce24c90e30f P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:50.080576 10152 raft_consensus.cc:374] T 96319242e7d6416bbe63cce24c90e30f P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:50.080613 10152 raft_consensus.cc:729] T 96319242e7d6416bbe63cce24c90e30f P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.080735 10152 consensus_queue.cc:260] T 96319242e7d6416bbe63cce24c90e30f P b0a54f7311df4dd2be068d3b15a4a360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:50.080989 10152 ts_tablet_manager.cc:1398] T 96319242e7d6416bbe63cce24c90e30f P b0a54f7311df4dd2be068d3b15a4a360: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:50.081107 10152 tablet_bootstrap.cc:492] T 6b48981997f145c581bf21555310d569 P b0a54f7311df4dd2be068d3b15a4a360: Bootstrap starting.
I20240502 13:59:50.081918 10152 tablet_bootstrap.cc:654] T 6b48981997f145c581bf21555310d569 P b0a54f7311df4dd2be068d3b15a4a360: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:50.082945 10156 tablet_bootstrap.cc:492] T 6b48981997f145c581bf21555310d569 P 4397244002fd4e0f9246b55727b536fe: No bootstrap required, opened a new log
I20240502 13:59:50.083039 10156 ts_tablet_manager.cc:1367] T 6b48981997f145c581bf21555310d569 P 4397244002fd4e0f9246b55727b536fe: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240502 13:59:50.083503 10152 tablet_bootstrap.cc:492] T 6b48981997f145c581bf21555310d569 P b0a54f7311df4dd2be068d3b15a4a360: No bootstrap required, opened a new log
I20240502 13:59:50.083596 10152 ts_tablet_manager.cc:1367] T 6b48981997f145c581bf21555310d569 P b0a54f7311df4dd2be068d3b15a4a360: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 13:59:50.083531 10156 raft_consensus.cc:348] T 6b48981997f145c581bf21555310d569 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:50.083638 10156 raft_consensus.cc:374] T 6b48981997f145c581bf21555310d569 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:50.083671 10156 raft_consensus.cc:729] T 6b48981997f145c581bf21555310d569 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.083807 10156 consensus_queue.cc:260] T 6b48981997f145c581bf21555310d569 P 4397244002fd4e0f9246b55727b536fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:50.084003 10152 raft_consensus.cc:348] T 6b48981997f145c581bf21555310d569 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:50.084091 10152 raft_consensus.cc:374] T 6b48981997f145c581bf21555310d569 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:50.084122 10152 raft_consensus.cc:729] T 6b48981997f145c581bf21555310d569 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.084225 10152 consensus_queue.cc:260] T 6b48981997f145c581bf21555310d569 P b0a54f7311df4dd2be068d3b15a4a360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:50.084502 10152 ts_tablet_manager.cc:1398] T 6b48981997f145c581bf21555310d569 P b0a54f7311df4dd2be068d3b15a4a360: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:50.084754 10156 ts_tablet_manager.cc:1398] T 6b48981997f145c581bf21555310d569 P 4397244002fd4e0f9246b55727b536fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:50.085446 10153 tablet_bootstrap.cc:492] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6: No bootstrap required, opened a new log
I20240502 13:59:50.085529 10153 ts_tablet_manager.cc:1367] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 13:59:50.085950 10153 raft_consensus.cc:348] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:50.086038 10153 raft_consensus.cc:374] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:50.086071 10153 raft_consensus.cc:729] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.086190 10153 consensus_queue.cc:260] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:50.086473 10153 ts_tablet_manager.cc:1398] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:50.119762 10088 raft_consensus.cc:482] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:50.119902 10088 raft_consensus.cc:504] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.120270 10088 leader_election.cc:288] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939), 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215)
I20240502 13:59:50.120708 9118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab480ba1dd34ca5b6694125374182dd" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541" is_pre_election: true
I20240502 13:59:50.120723 8855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab480ba1dd34ca5b6694125374182dd" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05" is_pre_election: true
I20240502 13:59:50.120849 9118 raft_consensus.cc:2454] T 9ab480ba1dd34ca5b6694125374182dd P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 0.
I20240502 13:59:50.120872 8855 raft_consensus.cc:2454] T 9ab480ba1dd34ca5b6694125374182dd P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 0.
I20240502 13:59:50.121183 9317 leader_election.cc:302] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:50.121397 10088 raft_consensus.cc:2781] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:50.121454 10088 raft_consensus.cc:482] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:50.121480 10088 raft_consensus.cc:3037] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:50.122359 10088 raft_consensus.cc:504] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.122650 10088 leader_election.cc:288] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 election: Requested vote from peers 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130:41939), 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215)
I20240502 13:59:50.122975 8855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab480ba1dd34ca5b6694125374182dd" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "379b5e0287244d6b94265ccffa51fc05"
I20240502 13:59:50.123049 9118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab480ba1dd34ca5b6694125374182dd" candidate_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541"
I20240502 13:59:50.123093 8855 raft_consensus.cc:3037] T 9ab480ba1dd34ca5b6694125374182dd P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:50.123163 9118 raft_consensus.cc:3037] T 9ab480ba1dd34ca5b6694125374182dd P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:50.123905 9118 raft_consensus.cc:2454] T 9ab480ba1dd34ca5b6694125374182dd P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 1.
I20240502 13:59:50.124058 8855 raft_consensus.cc:2454] T 9ab480ba1dd34ca5b6694125374182dd P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 833c2ad4636d4d8e9861339a4ad9e80e in term 1.
I20240502 13:59:50.124198 9317 leader_election.cc:302] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:50.124372 10088 raft_consensus.cc:2781] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:50.124459 10088 raft_consensus.cc:686] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 LEADER]: Becoming Leader. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Running, Role: LEADER
I20240502 13:59:50.124608 10088 consensus_queue.cc:237] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.126364 8572 catalog_manager.cc:5484] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e reported cstate change: term changed from 0 to 1, leader changed from <none> to 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134). New cstate: current_term: 1 leader_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:50.335865 10166 raft_consensus.cc:482] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:50.336010 10166 raft_consensus.cc:504] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:50.336508 10166 leader_election.cc:288] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699), 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573)
I20240502 13:59:50.336939 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96319242e7d6416bbe63cce24c90e30f" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360" is_pre_election: true
I20240502 13:59:50.337109 9251 raft_consensus.cc:2454] T 96319242e7d6416bbe63cce24c90e30f P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 0.
I20240502 13:59:50.337078 8722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96319242e7d6416bbe63cce24c90e30f" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe" is_pre_election: true
I20240502 13:59:50.337213 8722 raft_consensus.cc:2454] T 96319242e7d6416bbe63cce24c90e30f P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 0.
I20240502 13:59:50.337401 8922 leader_election.cc:302] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66ef9d15c0bd46fbb23d5ba11a085ba6, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:50.337565 10166 raft_consensus.cc:2781] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:50.337617 10166 raft_consensus.cc:482] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:50.337647 10166 raft_consensus.cc:3037] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:50.338543 10166 raft_consensus.cc:504] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:50.338857 10166 leader_election.cc:288] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 election: Requested vote from peers b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699), 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573)
I20240502 13:59:50.339200 8722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96319242e7d6416bbe63cce24c90e30f" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe"
I20240502 13:59:50.339197 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96319242e7d6416bbe63cce24c90e30f" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360"
I20240502 13:59:50.339303 8722 raft_consensus.cc:3037] T 96319242e7d6416bbe63cce24c90e30f P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:50.339323 9251 raft_consensus.cc:3037] T 96319242e7d6416bbe63cce24c90e30f P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:50.340049 9251 raft_consensus.cc:2454] T 96319242e7d6416bbe63cce24c90e30f P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 1.
I20240502 13:59:50.340101 8722 raft_consensus.cc:2454] T 96319242e7d6416bbe63cce24c90e30f P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 1.
I20240502 13:59:50.340389 8922 leader_election.cc:302] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66ef9d15c0bd46fbb23d5ba11a085ba6, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:50.340574 10166 raft_consensus.cc:2781] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:50.340672 10166 raft_consensus.cc:686] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 LEADER]: Becoming Leader. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Running, Role: LEADER
I20240502 13:59:50.340819 10166 consensus_queue.cc:237] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:50.342730 8572 catalog_manager.cc:5484] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131). New cstate: current_term: 1 leader_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:50.396533 10166 raft_consensus.cc:482] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:50.396664 10166 raft_consensus.cc:504] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:50.396978 10166 leader_election.cc:288] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699), 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573)
I20240502 13:59:50.397363 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b48981997f145c581bf21555310d569" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360" is_pre_election: true
I20240502 13:59:50.397363 8722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b48981997f145c581bf21555310d569" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe" is_pre_election: true
I20240502 13:59:50.397509 8722 raft_consensus.cc:2454] T 6b48981997f145c581bf21555310d569 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 0.
I20240502 13:59:50.397509 9251 raft_consensus.cc:2454] T 6b48981997f145c581bf21555310d569 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 0.
I20240502 13:59:50.397753 8922 leader_election.cc:302] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66ef9d15c0bd46fbb23d5ba11a085ba6, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:50.397938 10166 raft_consensus.cc:2781] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:50.398010 10166 raft_consensus.cc:482] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:50.398039 10166 raft_consensus.cc:3037] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:50.398797 10166 raft_consensus.cc:504] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:50.399073 10166 leader_election.cc:288] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 election: Requested vote from peers b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133:33699), 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573)
I20240502 13:59:50.399358 9251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b48981997f145c581bf21555310d569" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a54f7311df4dd2be068d3b15a4a360"
I20240502 13:59:50.399412 8722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b48981997f145c581bf21555310d569" candidate_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe"
I20240502 13:59:50.399504 8722 raft_consensus.cc:3037] T 6b48981997f145c581bf21555310d569 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:50.399497 9251 raft_consensus.cc:3037] T 6b48981997f145c581bf21555310d569 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:50.400381 8722 raft_consensus.cc:2454] T 6b48981997f145c581bf21555310d569 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 1.
I20240502 13:59:50.400381 9251 raft_consensus.cc:2454] T 6b48981997f145c581bf21555310d569 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66ef9d15c0bd46fbb23d5ba11a085ba6 in term 1.
I20240502 13:59:50.400655 8922 leader_election.cc:302] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66ef9d15c0bd46fbb23d5ba11a085ba6, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:50.400811 10166 raft_consensus.cc:2781] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:50.400910 10166 raft_consensus.cc:686] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 LEADER]: Becoming Leader. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Running, Role: LEADER
I20240502 13:59:50.401057 10166 consensus_queue.cc:237] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } }
I20240502 13:59:50.402709 8572 catalog_manager.cc:5484] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131). New cstate: current_term: 1 leader_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:50.550830 10224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:50.551018 10224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:50.553822 10224 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240502 13:59:50.591490 9988 consensus_queue.cc:1035] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:50.600318 10088 consensus_queue.cc:1035] T 9ab480ba1dd34ca5b6694125374182dd P 833c2ad4636d4d8e9861339a4ad9e80e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:50.647610 9251 raft_consensus.cc:1261] T 6b48981997f145c581bf21555310d569 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Refusing update from remote peer 66ef9d15c0bd46fbb23d5ba11a085ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:50.648170 9943 consensus_queue.cc:1035] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:50.649062 8722 raft_consensus.cc:1261] T 6b48981997f145c581bf21555310d569 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Refusing update from remote peer 66ef9d15c0bd46fbb23d5ba11a085ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:50.649602 9943 consensus_queue.cc:1035] T 6b48981997f145c581bf21555310d569 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:50.651749 9250 raft_consensus.cc:1261] T 96319242e7d6416bbe63cce24c90e30f P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Refusing update from remote peer 66ef9d15c0bd46fbb23d5ba11a085ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:50.651893 8723 raft_consensus.cc:1261] T 96319242e7d6416bbe63cce24c90e30f P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Refusing update from remote peer 66ef9d15c0bd46fbb23d5ba11a085ba6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:50.655875 10166 consensus_queue.cc:1035] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:50.656217 10166 consensus_queue.cc:1035] T 96319242e7d6416bbe63cce24c90e30f P 66ef9d15c0bd46fbb23d5ba11a085ba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 114.116 ms
time per row: 0.0570579 ms
I20240502 13:59:50.712476 8578 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54006:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:50.718115 8834 tablet_service.cc:1467] Processing CreateTablet for tablet 75aeb931a0984ad9acf42ed23f836caa (DEFAULT_TABLE table=rebalance_test_table_11 [id=6a4bce6b254247848ca03d00c735916e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:50.718266 9099 tablet_service.cc:1467] Processing CreateTablet for tablet 75aeb931a0984ad9acf42ed23f836caa (DEFAULT_TABLE table=rebalance_test_table_11 [id=6a4bce6b254247848ca03d00c735916e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:50.718472 8834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75aeb931a0984ad9acf42ed23f836caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.718555 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75aeb931a0984ad9acf42ed23f836caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.718843 9363 tablet_service.cc:1467] Processing CreateTablet for tablet 75aeb931a0984ad9acf42ed23f836caa (DEFAULT_TABLE table=rebalance_test_table_11 [id=6a4bce6b254247848ca03d00c735916e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:50.719173 9363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75aeb931a0984ad9acf42ed23f836caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.719461 8967 tablet_service.cc:1467] Processing CreateTablet for tablet 2718ec94ca8f46b0a61e4a2820ec8828 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6a4bce6b254247848ca03d00c735916e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:50.719712 8967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2718ec94ca8f46b0a61e4a2820ec8828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.720943 9230 tablet_service.cc:1467] Processing CreateTablet for tablet 2718ec94ca8f46b0a61e4a2820ec8828 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6a4bce6b254247848ca03d00c735916e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:50.721228 9230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2718ec94ca8f46b0a61e4a2820ec8828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.721570 8702 tablet_service.cc:1467] Processing CreateTablet for tablet 2718ec94ca8f46b0a61e4a2820ec8828 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6a4bce6b254247848ca03d00c735916e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:50.721837 8702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2718ec94ca8f46b0a61e4a2820ec8828. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.718843 9362 tablet_service.cc:1467] Processing CreateTablet for tablet 4d27765af0c04270a0160194020da65f (DEFAULT_TABLE table=rebalance_test_table_11 [id=6a4bce6b254247848ca03d00c735916e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:50.722477 9362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d27765af0c04270a0160194020da65f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.722582 8834 tablet_service.cc:1467] Processing CreateTablet for tablet 4d27765af0c04270a0160194020da65f (DEFAULT_TABLE table=rebalance_test_table_11 [id=6a4bce6b254247848ca03d00c735916e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:50.722796 8834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d27765af0c04270a0160194020da65f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.723825 9099 tablet_service.cc:1467] Processing CreateTablet for tablet 4d27765af0c04270a0160194020da65f (DEFAULT_TABLE table=rebalance_test_table_11 [id=6a4bce6b254247848ca03d00c735916e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:50.724076 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d27765af0c04270a0160194020da65f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:50.724092 10292 tablet_bootstrap.cc:492] T 75aeb931a0984ad9acf42ed23f836caa P 833c2ad4636d4d8e9861339a4ad9e80e: Bootstrap starting.
I20240502 13:59:50.725003 10292 tablet_bootstrap.cc:654] T 75aeb931a0984ad9acf42ed23f836caa P 833c2ad4636d4d8e9861339a4ad9e80e: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:50.726019 10293 tablet_bootstrap.cc:492] T 2718ec94ca8f46b0a61e4a2820ec8828 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Bootstrap starting.
I20240502 13:59:50.726425 10294 tablet_bootstrap.cc:492] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360: Bootstrap starting.
I20240502 13:59:50.726936 10293 tablet_bootstrap.cc:654] T 2718ec94ca8f46b0a61e4a2820ec8828 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:50.727617 10294 tablet_bootstrap.cc:654] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:50.727689 10291 tablet_bootstrap.cc:492] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05: Bootstrap starting.
I20240502 13:59:50.728085 10293 tablet_bootstrap.cc:492] T 2718ec94ca8f46b0a61e4a2820ec8828 P 66ef9d15c0bd46fbb23d5ba11a085ba6: No bootstrap required, opened a new log
I20240502 13:59:50.728189 10293 ts_tablet_manager.cc:1367] T 2718ec94ca8f46b0a61e4a2820ec8828 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:50.728565 10291 tablet_bootstrap.cc:654] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:50.728610 10294 tablet_bootstrap.cc:492] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360: No bootstrap required, opened a new log
I20240502 13:59:50.728689 10294 ts_tablet_manager.cc:1367] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:50.728674 10293 raft_consensus.cc:348] T 2718ec94ca8f46b0a61e4a2820ec8828 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } }
I20240502 13:59:50.728829 10293 raft_consensus.cc:374] T 2718ec94ca8f46b0a61e4a2820ec8828 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:50.728909 10293 raft_consensus.cc:729] T 2718ec94ca8f46b0a61e4a2820ec8828 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef9d15c0bd46fbb23d5ba11a085ba6, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.729099 10293 consensus_queue.cc:260] T 2718ec94ca8f46b0a61e4a2820ec8828 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } }
I20240502 13:59:50.729153 10294 raft_consensus.cc:348] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } }
I20240502 13:59:50.729229 10290 tablet_bootstrap.cc:492] T 75aeb931a0984ad9acf42ed23f836caa P 2695a61796d043aa823fe3ee966a7541: Bootstrap starting.
I20240502 13:59:50.729350 10294 raft_consensus.cc:374] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:50.729437 10294 raft_consensus.cc:729] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.729566 10293 ts_tablet_manager.cc:1398] T 2718ec94ca8f46b0a61e4a2820ec8828 P 66ef9d15c0bd46fbb23d5ba11a085ba6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:50.729615 10294 consensus_queue.cc:260] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } }
I20240502 13:59:50.729965 10291 tablet_bootstrap.cc:492] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05: No bootstrap required, opened a new log
I20240502 13:59:50.730098 10291 ts_tablet_manager.cc:1367] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:50.730614 10291 raft_consensus.cc:348] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.730751 10291 raft_consensus.cc:374] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:50.730783 10291 raft_consensus.cc:729] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.730916 10291 consensus_queue.cc:260] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.731204 10295 tablet_bootstrap.cc:492] T 2718ec94ca8f46b0a61e4a2820ec8828 P 4397244002fd4e0f9246b55727b536fe: Bootstrap starting.
I20240502 13:59:50.731704 10294 ts_tablet_manager.cc:1398] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 13:59:50.731930 10295 tablet_bootstrap.cc:654] T 2718ec94ca8f46b0a61e4a2820ec8828 P 4397244002fd4e0f9246b55727b536fe: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:50.732829 10290 tablet_bootstrap.cc:654] T 75aeb931a0984ad9acf42ed23f836caa P 2695a61796d043aa823fe3ee966a7541: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:50.733043 10295 tablet_bootstrap.cc:492] T 2718ec94ca8f46b0a61e4a2820ec8828 P 4397244002fd4e0f9246b55727b536fe: No bootstrap required, opened a new log
I20240502 13:59:50.733172 10295 ts_tablet_manager.cc:1367] T 2718ec94ca8f46b0a61e4a2820ec8828 P 4397244002fd4e0f9246b55727b536fe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:50.733199 10291 ts_tablet_manager.cc:1398] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 13:59:50.733371 10291 tablet_bootstrap.cc:492] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05: Bootstrap starting.
I20240502 13:59:50.733728 10295 raft_consensus.cc:348] T 2718ec94ca8f46b0a61e4a2820ec8828 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } }
I20240502 13:59:50.733901 10295 raft_consensus.cc:374] T 2718ec94ca8f46b0a61e4a2820ec8828 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:50.733974 10295 raft_consensus.cc:729] T 2718ec94ca8f46b0a61e4a2820ec8828 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4397244002fd4e0f9246b55727b536fe, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.734256 10291 tablet_bootstrap.cc:654] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:50.734189 10295 consensus_queue.cc:260] T 2718ec94ca8f46b0a61e4a2820ec8828 P 4397244002fd4e0f9246b55727b536fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } }
I20240502 13:59:50.735359 10295 ts_tablet_manager.cc:1398] T 2718ec94ca8f46b0a61e4a2820ec8828 P 4397244002fd4e0f9246b55727b536fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:50.735649 10291 tablet_bootstrap.cc:492] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05: No bootstrap required, opened a new log
I20240502 13:59:50.735728 10291 ts_tablet_manager.cc:1367] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:50.736076 10291 raft_consensus.cc:348] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.736166 10291 raft_consensus.cc:374] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:50.736486 10291 raft_consensus.cc:729] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.736662 10291 consensus_queue.cc:260] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.736997 10291 ts_tablet_manager.cc:1398] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:50.737048 10292 tablet_bootstrap.cc:492] T 75aeb931a0984ad9acf42ed23f836caa P 833c2ad4636d4d8e9861339a4ad9e80e: No bootstrap required, opened a new log
I20240502 13:59:50.737195 10292 ts_tablet_manager.cc:1367] T 75aeb931a0984ad9acf42ed23f836caa P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240502 13:59:50.737700 10292 raft_consensus.cc:348] T 75aeb931a0984ad9acf42ed23f836caa P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.737852 10292 raft_consensus.cc:374] T 75aeb931a0984ad9acf42ed23f836caa P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:50.737934 10292 raft_consensus.cc:729] T 75aeb931a0984ad9acf42ed23f836caa P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.737967 10290 tablet_bootstrap.cc:492] T 75aeb931a0984ad9acf42ed23f836caa P 2695a61796d043aa823fe3ee966a7541: No bootstrap required, opened a new log
I20240502 13:59:50.738126 10290 ts_tablet_manager.cc:1367] T 75aeb931a0984ad9acf42ed23f836caa P 2695a61796d043aa823fe3ee966a7541: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 13:59:50.738099 10292 consensus_queue.cc:260] T 75aeb931a0984ad9acf42ed23f836caa P 833c2ad4636d4d8e9861339a4ad9e80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.738417 10292 ts_tablet_manager.cc:1398] T 75aeb931a0984ad9acf42ed23f836caa P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:50.738584 10290 raft_consensus.cc:348] T 75aeb931a0984ad9acf42ed23f836caa P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.738734 10290 raft_consensus.cc:374] T 75aeb931a0984ad9acf42ed23f836caa P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:50.738780 10290 raft_consensus.cc:729] T 75aeb931a0984ad9acf42ed23f836caa P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.739110 10292 tablet_bootstrap.cc:492] T 4d27765af0c04270a0160194020da65f P 833c2ad4636d4d8e9861339a4ad9e80e: Bootstrap starting.
I20240502 13:59:50.739125 10290 consensus_queue.cc:260] T 75aeb931a0984ad9acf42ed23f836caa P 2695a61796d043aa823fe3ee966a7541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.739455 10290 ts_tablet_manager.cc:1398] T 75aeb931a0984ad9acf42ed23f836caa P 2695a61796d043aa823fe3ee966a7541: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:50.739670 10290 tablet_bootstrap.cc:492] T 4d27765af0c04270a0160194020da65f P 2695a61796d043aa823fe3ee966a7541: Bootstrap starting.
I20240502 13:59:50.740375 10292 tablet_bootstrap.cc:654] T 4d27765af0c04270a0160194020da65f P 833c2ad4636d4d8e9861339a4ad9e80e: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:50.740839 10290 tablet_bootstrap.cc:654] T 4d27765af0c04270a0160194020da65f P 2695a61796d043aa823fe3ee966a7541: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:50.741904 10292 tablet_bootstrap.cc:492] T 4d27765af0c04270a0160194020da65f P 833c2ad4636d4d8e9861339a4ad9e80e: No bootstrap required, opened a new log
I20240502 13:59:50.742063 10292 ts_tablet_manager.cc:1367] T 4d27765af0c04270a0160194020da65f P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 13:59:50.742151 10290 tablet_bootstrap.cc:492] T 4d27765af0c04270a0160194020da65f P 2695a61796d043aa823fe3ee966a7541: No bootstrap required, opened a new log
I20240502 13:59:50.742259 10290 ts_tablet_manager.cc:1367] T 4d27765af0c04270a0160194020da65f P 2695a61796d043aa823fe3ee966a7541: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 13:59:50.742566 10292 raft_consensus.cc:348] T 4d27765af0c04270a0160194020da65f P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.742681 10292 raft_consensus.cc:374] T 4d27765af0c04270a0160194020da65f P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:50.742718 10292 raft_consensus.cc:729] T 4d27765af0c04270a0160194020da65f P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833c2ad4636d4d8e9861339a4ad9e80e, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.742859 10290 raft_consensus.cc:348] T 4d27765af0c04270a0160194020da65f P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.742871 10292 consensus_queue.cc:260] T 4d27765af0c04270a0160194020da65f P 833c2ad4636d4d8e9861339a4ad9e80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.743021 10290 raft_consensus.cc:374] T 4d27765af0c04270a0160194020da65f P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:50.743098 10290 raft_consensus.cc:729] T 4d27765af0c04270a0160194020da65f P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2695a61796d043aa823fe3ee966a7541, State: Initialized, Role: FOLLOWER
I20240502 13:59:50.743208 10292 ts_tablet_manager.cc:1398] T 4d27765af0c04270a0160194020da65f P 833c2ad4636d4d8e9861339a4ad9e80e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:50.743304 10290 consensus_queue.cc:260] T 4d27765af0c04270a0160194020da65f P 2695a61796d043aa823fe3ee966a7541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.743937 10290 ts_tablet_manager.cc:1398] T 4d27765af0c04270a0160194020da65f P 2695a61796d043aa823fe3ee966a7541: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:50.832231 9722 raft_consensus.cc:482] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:50.832355 9722 raft_consensus.cc:504] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } }
I20240502 13:59:50.832804 9722 leader_election.cc:288] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573), 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873)
I20240502 13:59:50.833112 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2718ec94ca8f46b0a61e4a2820ec8828" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe" is_pre_election: true
I20240502 13:59:50.833166 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2718ec94ca8f46b0a61e4a2820ec8828" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" is_pre_election: true
I20240502 13:59:50.833245 8723 raft_consensus.cc:2454] T 2718ec94ca8f46b0a61e4a2820ec8828 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 0.
I20240502 13:59:50.833314 8987 raft_consensus.cc:2454] T 2718ec94ca8f46b0a61e4a2820ec8828 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 0.
I20240502 13:59:50.833547 9185 leader_election.cc:302] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:50.833700 9722 raft_consensus.cc:2781] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:50.833760 9722 raft_consensus.cc:482] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:50.833786 9722 raft_consensus.cc:3037] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:50.834645 9722 raft_consensus.cc:504] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } }
I20240502 13:59:50.835003 9722 leader_election.cc:288] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 election: Requested vote from peers 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573), 66ef9d15c0bd46fbb23d5ba11a085ba6 (127.8.86.131:39873)
I20240502 13:59:50.835279 8723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2718ec94ca8f46b0a61e4a2820ec8828" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4397244002fd4e0f9246b55727b536fe"
I20240502 13:59:50.835299 8987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2718ec94ca8f46b0a61e4a2820ec8828" candidate_uuid: "b0a54f7311df4dd2be068d3b15a4a360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6"
I20240502 13:59:50.835386 8987 raft_consensus.cc:3037] T 2718ec94ca8f46b0a61e4a2820ec8828 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:50.835402 8723 raft_consensus.cc:3037] T 2718ec94ca8f46b0a61e4a2820ec8828 P 4397244002fd4e0f9246b55727b536fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:50.836083 8723 raft_consensus.cc:2454] T 2718ec94ca8f46b0a61e4a2820ec8828 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 1.
I20240502 13:59:50.836156 8987 raft_consensus.cc:2454] T 2718ec94ca8f46b0a61e4a2820ec8828 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0a54f7311df4dd2be068d3b15a4a360 in term 1.
I20240502 13:59:50.836369 9185 leader_election.cc:302] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4397244002fd4e0f9246b55727b536fe, b0a54f7311df4dd2be068d3b15a4a360; no voters:
I20240502 13:59:50.836571 9722 raft_consensus.cc:2781] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:50.836652 9722 raft_consensus.cc:686] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 [term 1 LEADER]: Becoming Leader. State: Replica: b0a54f7311df4dd2be068d3b15a4a360, State: Running, Role: LEADER
I20240502 13:59:50.836805 9722 consensus_queue.cc:237] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } }
I20240502 13:59:50.838701 8572 catalog_manager.cc:5484] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0a54f7311df4dd2be068d3b15a4a360 (127.8.86.133). New cstate: current_term: 1 leader_uuid: "b0a54f7311df4dd2be068d3b15a4a360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a54f7311df4dd2be068d3b15a4a360" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 33699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:50.905772 10001 raft_consensus.cc:482] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:50.905890 10001 raft_consensus.cc:504] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.906281 10001 leader_election.cc:288] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134:35071)
I20240502 13:59:50.906714 9118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d27765af0c04270a0160194020da65f" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541" is_pre_election: true
I20240502 13:59:50.906750 9383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d27765af0c04270a0160194020da65f" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" is_pre_election: true
I20240502 13:59:50.906867 9383 raft_consensus.cc:2454] T 4d27765af0c04270a0160194020da65f P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 0.
I20240502 13:59:50.906867 9118 raft_consensus.cc:2454] T 4d27765af0c04270a0160194020da65f P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 0.
I20240502 13:59:50.907157 8789 leader_election.cc:302] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 379b5e0287244d6b94265ccffa51fc05; no voters:
I20240502 13:59:50.907332 10306 raft_consensus.cc:2781] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:50.907476 10306 raft_consensus.cc:482] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:50.907542 10306 raft_consensus.cc:3037] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:50.908368 10306 raft_consensus.cc:504] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.908737 10306 leader_election.cc:288] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 election: Requested vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134:35071)
I20240502 13:59:50.909014 9382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d27765af0c04270a0160194020da65f" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833c2ad4636d4d8e9861339a4ad9e80e"
I20240502 13:59:50.909036 9118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d27765af0c04270a0160194020da65f" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541"
I20240502 13:59:50.909118 9382 raft_consensus.cc:3037] T 4d27765af0c04270a0160194020da65f P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:50.909161 9118 raft_consensus.cc:3037] T 4d27765af0c04270a0160194020da65f P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:50.909945 9382 raft_consensus.cc:2454] T 4d27765af0c04270a0160194020da65f P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 1.
I20240502 13:59:50.909998 9118 raft_consensus.cc:2454] T 4d27765af0c04270a0160194020da65f P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 1.
I20240502 13:59:50.910241 8790 leader_election.cc:302] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 379b5e0287244d6b94265ccffa51fc05, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:50.910442 10306 raft_consensus.cc:2781] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:50.910530 10306 raft_consensus.cc:686] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 [term 1 LEADER]: Becoming Leader. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Running, Role: LEADER
I20240502 13:59:50.910667 10306 consensus_queue.cc:237] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.912477 8578 catalog_manager.cc:5484] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "379b5e0287244d6b94265ccffa51fc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:50.914714 10306 raft_consensus.cc:482] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:50.914793 10306 raft_consensus.cc:504] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.915052 10306 leader_election.cc:288] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134:35071)
I20240502 13:59:50.915349 9118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75aeb931a0984ad9acf42ed23f836caa" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541" is_pre_election: true
I20240502 13:59:50.915397 9382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75aeb931a0984ad9acf42ed23f836caa" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" is_pre_election: true
I20240502 13:59:50.915472 9118 raft_consensus.cc:2454] T 75aeb931a0984ad9acf42ed23f836caa P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 0.
I20240502 13:59:50.915506 9382 raft_consensus.cc:2454] T 75aeb931a0984ad9acf42ed23f836caa P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 0.
I20240502 13:59:50.915716 8790 leader_election.cc:302] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 379b5e0287244d6b94265ccffa51fc05, 833c2ad4636d4d8e9861339a4ad9e80e; no voters:
I20240502 13:59:50.915903 10306 raft_consensus.cc:2781] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:50.915959 10306 raft_consensus.cc:482] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:50.916002 10306 raft_consensus.cc:3037] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:50.916644 10306 raft_consensus.cc:504] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.916936 10306 leader_election.cc:288] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 election: Requested vote from peers 2695a61796d043aa823fe3ee966a7541 (127.8.86.132:35215), 833c2ad4636d4d8e9861339a4ad9e80e (127.8.86.134:35071)
I20240502 13:59:50.917210 9118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75aeb931a0984ad9acf42ed23f836caa" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2695a61796d043aa823fe3ee966a7541"
I20240502 13:59:50.917300 9118 raft_consensus.cc:3037] T 75aeb931a0984ad9acf42ed23f836caa P 2695a61796d043aa823fe3ee966a7541 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:50.917922 9118 raft_consensus.cc:2454] T 75aeb931a0984ad9acf42ed23f836caa P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 1.
I20240502 13:59:50.918113 9382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75aeb931a0984ad9acf42ed23f836caa" candidate_uuid: "379b5e0287244d6b94265ccffa51fc05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "833c2ad4636d4d8e9861339a4ad9e80e"
I20240502 13:59:50.918190 8789 leader_election.cc:302] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2695a61796d043aa823fe3ee966a7541, 379b5e0287244d6b94265ccffa51fc05; no voters:
I20240502 13:59:50.918218 9382 raft_consensus.cc:3037] T 75aeb931a0984ad9acf42ed23f836caa P 833c2ad4636d4d8e9861339a4ad9e80e [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:50.918383 10306 raft_consensus.cc:2781] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:50.918473 10306 raft_consensus.cc:686] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 [term 1 LEADER]: Becoming Leader. State: Replica: 379b5e0287244d6b94265ccffa51fc05, State: Running, Role: LEADER
I20240502 13:59:50.918594 10306 consensus_queue.cc:237] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } }
I20240502 13:59:50.919024 9382 raft_consensus.cc:2454] T 75aeb931a0984ad9acf42ed23f836caa P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 379b5e0287244d6b94265ccffa51fc05 in term 1.
I20240502 13:59:50.920433 8578 catalog_manager.cc:5484] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 379b5e0287244d6b94265ccffa51fc05 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "379b5e0287244d6b94265ccffa51fc05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "379b5e0287244d6b94265ccffa51fc05" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 41939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:51.064049 10309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:51.064285 10309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:51.067718 10309 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240502 13:59:51.158232 9382 raft_consensus.cc:1261] T 75aeb931a0984ad9acf42ed23f836caa P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Refusing update from remote peer 379b5e0287244d6b94265ccffa51fc05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:51.159216 9118 raft_consensus.cc:1261] T 4d27765af0c04270a0160194020da65f P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Refusing update from remote peer 379b5e0287244d6b94265ccffa51fc05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:51.159312 10306 consensus_queue.cc:1035] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:51.159826 10306 consensus_queue.cc:1035] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:51.160544 9118 raft_consensus.cc:1261] T 75aeb931a0984ad9acf42ed23f836caa P 2695a61796d043aa823fe3ee966a7541 [term 1 FOLLOWER]: Refusing update from remote peer 379b5e0287244d6b94265ccffa51fc05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:51.161101 9382 raft_consensus.cc:1261] T 4d27765af0c04270a0160194020da65f P 833c2ad4636d4d8e9861339a4ad9e80e [term 1 FOLLOWER]: Refusing update from remote peer 379b5e0287244d6b94265ccffa51fc05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:51.161546 10001 consensus_queue.cc:1035] T 75aeb931a0984ad9acf42ed23f836caa P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2695a61796d043aa823fe3ee966a7541" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:51.161826 10001 consensus_queue.cc:1035] T 4d27765af0c04270a0160194020da65f P 379b5e0287244d6b94265ccffa51fc05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "833c2ad4636d4d8e9861339a4ad9e80e" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 35071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:51.162391 8987 raft_consensus.cc:1261] T 2718ec94ca8f46b0a61e4a2820ec8828 P 66ef9d15c0bd46fbb23d5ba11a085ba6 [term 1 FOLLOWER]: Refusing update from remote peer b0a54f7311df4dd2be068d3b15a4a360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:51.164374 10146 consensus_queue.cc:1035] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66ef9d15c0bd46fbb23d5ba11a085ba6" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:51.166661 8723 raft_consensus.cc:1261] T 2718ec94ca8f46b0a61e4a2820ec8828 P 4397244002fd4e0f9246b55727b536fe [term 1 FOLLOWER]: Refusing update from remote peer b0a54f7311df4dd2be068d3b15a4a360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:51.167204 10146 consensus_queue.cc:1035] T 2718ec94ca8f46b0a61e4a2820ec8828 P b0a54f7311df4dd2be068d3b15a4a360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4397244002fd4e0f9246b55727b536fe" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 41573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 115.093 ms
time per row: 0.0575465 ms
I20240502 13:59:51.445052 8834 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:51.445757 8967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 13:59:51.446425 8702 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:51.447547 9099 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:51.449506 9230 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:51.468454 9362 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 13:59:51.579906 9362 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 13:59:51.581914 9099 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:51.583235 9230 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:51.585224 8967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 13:59:51.585469 8834 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:51.594493 8702 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:51.741057 9099 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:51.741748 8834 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:51.745873 9362 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 13:59:51.746374 9230 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:51.751255 8967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 13:59:51.752985 8702 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:51.868160 8702 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:51.870873 8834 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:51.871304 9099 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:51.871583 8967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 13:59:51.874382 9362 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 13:59:51.881364 9230 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:52.001953 9099 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:52.006345 8967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 13:59:52.007447 8834 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:52.018574 9362 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 13:59:52.028084 9230 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:52.034255 8702 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:52.148974 9099 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:52.150224 8834 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:52.150677 8967 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 13:59:52.151439 9362 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 13:59:52.153499 8702 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:52.169030 9230 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
65d1b70050584073b6a58e6ef213ef9b | 127.8.86.190:40597 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.8.86.148:33373 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
2695a61796d043aa823fe3ee966a7541 | 127.8.86.132:35215 | HEALTHY | /B | 2 | 0
379b5e0287244d6b94265ccffa51fc05 | 127.8.86.130:41939 | HEALTHY | /A | 7 | 0
4397244002fd4e0f9246b55727b536fe | 127.8.86.129:41573 | HEALTHY | /A | 5 | 0
66ef9d15c0bd46fbb23d5ba11a085ba6 | 127.8.86.131:39873 | HEALTHY | /B | 8 | 0
833c2ad4636d4d8e9861339a4ad9e80e | 127.8.86.134:35071 | HEALTHY | /C | 9 | 0
b0a54f7311df4dd2be068d3b15a4a360 | 127.8.86.133:33699 | HEALTHY | /C | 5 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/C | 2
/B | 2
/A | 2
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.8.86.129 | experimental | 127.8.86.129:41573
local_ip_for_outbound_sockets | 127.8.86.130 | experimental | 127.8.86.130:41939
local_ip_for_outbound_sockets | 127.8.86.131 | experimental | 127.8.86.131:39873
local_ip_for_outbound_sockets | 127.8.86.132 | experimental | 127.8.86.132:35215
local_ip_for_outbound_sockets | 127.8.86.133 | experimental | 127.8.86.133:33699
local_ip_for_outbound_sockets | 127.8.86.134 | experimental | 127.8.86.134:35071
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.8.86.129:41573
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.8.86.130:41939
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.8.86.131:39873
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.8.86.132:35215
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.8.86.133:33699
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.8.86.134:35071
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.8.86.148:33373 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_10 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_11 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 18
Median | 18
Third Quartile | 18
Maximum | 18
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 12
Tablets | 36
Replicas | 108
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 13:59:52.205205 8538 log_verifier.cc:126] Checking tablet 16ec7874224d4b709d2df8f893f41678
I20240502 13:59:52.234895 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 16ec7874224d4b709d2df8f893f41678
I20240502 13:59:52.235002 8538 log_verifier.cc:126] Checking tablet 16eeb3fe7ac34432b9dce18727289e1c
I20240502 13:59:52.261340 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 16eeb3fe7ac34432b9dce18727289e1c
I20240502 13:59:52.261415 8538 log_verifier.cc:126] Checking tablet 205cc772d7e44e289670490fb705dff8
I20240502 13:59:52.286554 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 205cc772d7e44e289670490fb705dff8
I20240502 13:59:52.286635 8538 log_verifier.cc:126] Checking tablet 2718ec94ca8f46b0a61e4a2820ec8828
I20240502 13:59:52.311962 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2718ec94ca8f46b0a61e4a2820ec8828
I20240502 13:59:52.312027 8538 log_verifier.cc:126] Checking tablet 3333dcab7e814b42ae88630f83f6ed84
I20240502 13:59:52.335886 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3333dcab7e814b42ae88630f83f6ed84
I20240502 13:59:52.335954 8538 log_verifier.cc:126] Checking tablet 35debb019a2940dfb72d52fcb8c2ddbc
I20240502 13:59:52.361274 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 35debb019a2940dfb72d52fcb8c2ddbc
I20240502 13:59:52.361346 8538 log_verifier.cc:126] Checking tablet 3836daa0c16c41688432d5cf816eaff9
I20240502 13:59:52.386991 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3836daa0c16c41688432d5cf816eaff9
I20240502 13:59:52.387056 8538 log_verifier.cc:126] Checking tablet 3a034d01e1b5402681c95d135646b88f
I20240502 13:59:52.412667 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3a034d01e1b5402681c95d135646b88f
I20240502 13:59:52.412732 8538 log_verifier.cc:126] Checking tablet 3c45225f7cfa4821a6f75abc2dd0d43d
I20240502 13:59:52.437371 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3c45225f7cfa4821a6f75abc2dd0d43d
I20240502 13:59:52.437443 8538 log_verifier.cc:126] Checking tablet 4368b81254a24d1cba21912300deb7db
I20240502 13:59:52.462654 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4368b81254a24d1cba21912300deb7db
I20240502 13:59:52.462724 8538 log_verifier.cc:126] Checking tablet 49e3031937504d989e9a63115d8f9c07
I20240502 13:59:52.487205 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 49e3031937504d989e9a63115d8f9c07
I20240502 13:59:52.487270 8538 log_verifier.cc:126] Checking tablet 4d27765af0c04270a0160194020da65f
I20240502 13:59:52.513592 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4d27765af0c04270a0160194020da65f
I20240502 13:59:52.513679 8538 log_verifier.cc:126] Checking tablet 4f1a21869b924bc79a6f82d5d0952dd7
I20240502 13:59:52.541102 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4f1a21869b924bc79a6f82d5d0952dd7
I20240502 13:59:52.541189 8538 log_verifier.cc:126] Checking tablet 58d1f57210e04e1482004fe770479d7e
I20240502 13:59:52.567106 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 58d1f57210e04e1482004fe770479d7e
I20240502 13:59:52.567169 8538 log_verifier.cc:126] Checking tablet 5b1f2054645e4f2bb2e26831f698e871
I20240502 13:59:52.591871 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5b1f2054645e4f2bb2e26831f698e871
I20240502 13:59:52.591953 8538 log_verifier.cc:126] Checking tablet 6155c2fe97bd41b59314d94e4c3f2a44
I20240502 13:59:52.623445 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6155c2fe97bd41b59314d94e4c3f2a44
I20240502 13:59:52.623507 8538 log_verifier.cc:126] Checking tablet 66671d0baed54d17bc6f3ce1535d8550
I20240502 13:59:52.651389 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 66671d0baed54d17bc6f3ce1535d8550
I20240502 13:59:52.651454 8538 log_verifier.cc:126] Checking tablet 6b48981997f145c581bf21555310d569
I20240502 13:59:52.676235 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6b48981997f145c581bf21555310d569
I20240502 13:59:52.676308 8538 log_verifier.cc:126] Checking tablet 72e38e431ec0424ba8b1e7813e26aa27
I20240502 13:59:52.701483 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 72e38e431ec0424ba8b1e7813e26aa27
I20240502 13:59:52.701575 8538 log_verifier.cc:126] Checking tablet 75aeb931a0984ad9acf42ed23f836caa
I20240502 13:59:52.727002 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 75aeb931a0984ad9acf42ed23f836caa
I20240502 13:59:52.727073 8538 log_verifier.cc:126] Checking tablet 8a3778c753514d17965f2f1dbe0a418a
I20240502 13:59:52.751685 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8a3778c753514d17965f2f1dbe0a418a
I20240502 13:59:52.751753 8538 log_verifier.cc:126] Checking tablet 96319242e7d6416bbe63cce24c90e30f
I20240502 13:59:52.776165 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 96319242e7d6416bbe63cce24c90e30f
I20240502 13:59:52.776226 8538 log_verifier.cc:126] Checking tablet 96f107cb7739454f906466f5c061d3fc
I20240502 13:59:52.801244 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 96f107cb7739454f906466f5c061d3fc
I20240502 13:59:52.801314 8538 log_verifier.cc:126] Checking tablet 9ab480ba1dd34ca5b6694125374182dd
I20240502 13:59:52.826413 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9ab480ba1dd34ca5b6694125374182dd
I20240502 13:59:52.826486 8538 log_verifier.cc:126] Checking tablet 9cb54b4795434871aa9530fc002f0beb
I20240502 13:59:52.850936 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9cb54b4795434871aa9530fc002f0beb
I20240502 13:59:52.851009 8538 log_verifier.cc:126] Checking tablet 9f39cfbee042492cb621d900a7fdbc8b
I20240502 13:59:52.875568 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9f39cfbee042492cb621d900a7fdbc8b
I20240502 13:59:52.875646 8538 log_verifier.cc:126] Checking tablet b0012eb13b8b49c4a8ebd0fbf5bc70be
I20240502 13:59:52.901049 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b0012eb13b8b49c4a8ebd0fbf5bc70be
I20240502 13:59:52.901124 8538 log_verifier.cc:126] Checking tablet b45b34b813a745eb9864caab8e44362e
I20240502 13:59:52.926539 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b45b34b813a745eb9864caab8e44362e
I20240502 13:59:52.926620 8538 log_verifier.cc:126] Checking tablet badff6c7db104260924db424787cac2a
I20240502 13:59:52.951361 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet badff6c7db104260924db424787cac2a
I20240502 13:59:52.951440 8538 log_verifier.cc:126] Checking tablet bdae32d952674a2796d571db6b00f839
I20240502 13:59:52.976249 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet bdae32d952674a2796d571db6b00f839
I20240502 13:59:52.976311 8538 log_verifier.cc:126] Checking tablet bfa322f449a64cf69ebafc72bca32faa
I20240502 13:59:53.001236 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet bfa322f449a64cf69ebafc72bca32faa
I20240502 13:59:53.001307 8538 log_verifier.cc:126] Checking tablet bfe2461e26cf47c489b111af958e9309
I20240502 13:59:53.026324 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet bfe2461e26cf47c489b111af958e9309
I20240502 13:59:53.026396 8538 log_verifier.cc:126] Checking tablet e1f07eacd9af4363a0506ad9ebace9fd
I20240502 13:59:53.051394 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e1f07eacd9af4363a0506ad9ebace9fd
I20240502 13:59:53.051465 8538 log_verifier.cc:126] Checking tablet ec4945a457e245c992507afb398d7287
I20240502 13:59:53.075978 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ec4945a457e245c992507afb398d7287
I20240502 13:59:53.076049 8538 log_verifier.cc:126] Checking tablet f093a86fa3ea4742bd70344f6f183293
I20240502 13:59:53.100730 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f093a86fa3ea4742bd70344f6f183293
I20240502 13:59:53.100791 8538 log_verifier.cc:126] Checking tablet fcf2b394ee794547824377bc3a4eae9a
I20240502 13:59:53.124622 8538 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fcf2b394ee794547824377bc3a4eae9a
I20240502 13:59:53.137728 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 8641
I20240502 13:59:53.145268 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 8772
W20240502 13:59:53.147377 8921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.129:41573: connect: Connection refused (error 111)
W20240502 13:59:53.148252 8921 consensus_peers.cc:485] T 3a034d01e1b5402681c95d135646b88f P 66ef9d15c0bd46fbb23d5ba11a085ba6 -> Peer 4397244002fd4e0f9246b55727b536fe (127.8.86.129:41573): Couldn't send request to peer 4397244002fd4e0f9246b55727b536fe. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.129:41573: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:53.152426 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 8904
I20240502 13:59:53.159335 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 9036
I20240502 13:59:53.165674 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 9168
I20240502 13:59:53.171766 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 9300
I20240502 13:59:53.177887 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 8550
2024-05-02T13:59:53Z chronyd exiting
[ OK ] LocationAwareRebalancingBasicTest.Basic (14125 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (14125 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20240502 13:59:53.296572 8538 test_util.cc:268] Using random seed: 74550966
I20240502 13:59:53.297410 8538 ts_itest-base.cc:112] Starting cluster with:
I20240502 13:59:53.297448 8538 ts_itest-base.cc:113] --------------
I20240502 13:59:53.297461 8538 ts_itest-base.cc:114] 5 tablet servers
I20240502 13:59:53.297478 8538 ts_itest-base.cc:115] 3 replicas per TS
I20240502 13:59:53.297500 8538 ts_itest-base.cc:116] --------------
2024-05-02T13:59:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T13:59:53Z Disabled control of system clock
I20240502 13:59:53.305461 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.86.190:44391
--webserver_interface=127.8.86.190
--webserver_port=0
--builtin_ntp_servers=127.8.86.148:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.86.190:44391
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 13:59:53.406569 10641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:53.406790 10641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:53.406852 10641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:53.409651 10641 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:53.409718 10641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 13:59:53.409765 10641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:53.409809 10641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 13:59:53.409842 10641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 13:59:53.413185 10641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.8.86.190:44391
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.86.190:44391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.8.86.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.10641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:53.413981 10641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:53.415014 10641 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:53.422428 10647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:53.423161 10649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:53.423812 10646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:53.424605 10641 server_base.cc:1034] running on GCE node
I20240502 13:59:53.425071 10641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:53.425925 10641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:53.427060 10641 hybrid_clock.cc:648] HybridClock initialized: now 1714658393427050 us; error 39 us; skew 500 ppm
I20240502 13:59:53.428758 10641 webserver.cc:460] Webserver started at http://127.8.86.190:38439/ using document root <none> and password file <none>
I20240502 13:59:53.429257 10641 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:53.429309 10641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:53.429482 10641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:53.431046 10641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9373da715c7142a9a740c23881429147"
format_stamp: "Formatted at 2024-05-02 13:59:53 on dist-test-slave-h6v0"
I20240502 13:59:53.431488 10641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9373da715c7142a9a740c23881429147"
format_stamp: "Formatted at 2024-05-02 13:59:53 on dist-test-slave-h6v0"
I20240502 13:59:53.434484 10641 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20240502 13:59:53.436431 10655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:53.437340 10641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:53.437451 10641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9373da715c7142a9a740c23881429147"
format_stamp: "Formatted at 2024-05-02 13:59:53 on dist-test-slave-h6v0"
I20240502 13:59:53.437526 10641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:53.449621 10641 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:53.450178 10641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:53.450349 10641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:53.456346 10641 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.190:44391
I20240502 13:59:53.456357 10707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.190:44391 every 8 connection(s)
I20240502 13:59:53.457263 10641 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 13:59:53.459654 10708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:53.460160 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 10641
I20240502 13:59:53.460280 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 13:59:53.465081 10708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147: Bootstrap starting.
I20240502 13:59:53.467442 10708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:53.468209 10708 log.cc:824] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:53.469821 10708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147: No bootstrap required, opened a new log
I20240502 13:59:53.472101 10708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9373da715c7142a9a740c23881429147" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 44391 } }
I20240502 13:59:53.472275 10708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:53.472312 10708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9373da715c7142a9a740c23881429147, State: Initialized, Role: FOLLOWER
I20240502 13:59:53.472654 10708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9373da715c7142a9a740c23881429147" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 44391 } }
I20240502 13:59:53.472759 10708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:53.472798 10708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:53.472867 10708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:53.473469 10708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9373da715c7142a9a740c23881429147" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 44391 } }
I20240502 13:59:53.473721 10708 leader_election.cc:302] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9373da715c7142a9a740c23881429147; no voters:
I20240502 13:59:53.473942 10708 leader_election.cc:288] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:53.474078 10713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:53.474326 10713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147 [term 1 LEADER]: Becoming Leader. State: Replica: 9373da715c7142a9a740c23881429147, State: Running, Role: LEADER
I20240502 13:59:53.474601 10713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9373da715c7142a9a740c23881429147" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 44391 } }
I20240502 13:59:53.475070 10708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 13:59:53.476333 10715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9373da715c7142a9a740c23881429147. Latest consensus state: current_term: 1 leader_uuid: "9373da715c7142a9a740c23881429147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9373da715c7142a9a740c23881429147" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 44391 } } }
I20240502 13:59:53.476473 10715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147 [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:53.476361 10714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9373da715c7142a9a740c23881429147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9373da715c7142a9a740c23881429147" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 44391 } } }
I20240502 13:59:53.476621 10714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147 [sys.catalog]: This master's current role is: LEADER
W20240502 13:59:53.479714 10729 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 13:59:53.479776 10729 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 13:59:53.479846 10721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 13:59:53.480561 10721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 13:59:53.484787 10721 catalog_manager.cc:1347] Generated new cluster ID: 487ae0a5a3b24e9c83997c6f3ac1b09d
I20240502 13:59:53.484849 10721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 13:59:53.498732 10721 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 13:59:53.499568 10721 catalog_manager.cc:1504] Loading token signing keys...
I20240502 13:59:53.514703 10721 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9373da715c7142a9a740c23881429147: Generated new TSK 0
I20240502 13:59:53.515388 10721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 13:59:53.520681 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.129:0
--local_ip_for_outbound_sockets=127.8.86.129
--webserver_interface=127.8.86.129
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44391
--builtin_ntp_servers=127.8.86.148:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 13:59:53.627619 10732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:53.627823 10732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:53.627871 10732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:53.630923 10732 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:53.630983 10732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:53.631067 10732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.129
I20240502 13:59:53.634650 10732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.8.86.129
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44391
--never_fsync=true
--heap_profile_path=/tmp/kudu.10732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.129
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:53.635500 10732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:53.636595 10732 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:53.644981 10737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:53.645494 10740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:53.646317 10738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:53.646996 10732 server_base.cc:1034] running on GCE node
I20240502 13:59:53.647408 10732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:53.647949 10732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:53.649109 10732 hybrid_clock.cc:648] HybridClock initialized: now 1714658393649095 us; error 40 us; skew 500 ppm
I20240502 13:59:53.651067 10732 webserver.cc:460] Webserver started at http://127.8.86.129:46607/ using document root <none> and password file <none>
I20240502 13:59:53.651618 10732 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:53.651679 10732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:53.651847 10732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:53.653450 10732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "693080b78f6941119ae0404d7b10e687"
format_stamp: "Formatted at 2024-05-02 13:59:53 on dist-test-slave-h6v0"
I20240502 13:59:53.653918 10732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "693080b78f6941119ae0404d7b10e687"
format_stamp: "Formatted at 2024-05-02 13:59:53 on dist-test-slave-h6v0"
I20240502 13:59:53.657086 10732 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 13:59:53.659207 10746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:53.660195 10732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:53.660301 10732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "693080b78f6941119ae0404d7b10e687"
format_stamp: "Formatted at 2024-05-02 13:59:53 on dist-test-slave-h6v0"
I20240502 13:59:53.660384 10732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:53.672470 10732 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:53.673012 10732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:53.673166 10732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:53.673658 10732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:53.674592 10732 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:53.674643 10732 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:53.674711 10732 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:53.674736 10732 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:53.683277 10732 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.129:34875
I20240502 13:59:53.683316 10859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.129:34875 every 8 connection(s)
I20240502 13:59:53.684173 10732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 13:59:53.686252 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 10732
I20240502 13:59:53.686395 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 13:59:53.688536 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.130:0
--local_ip_for_outbound_sockets=127.8.86.130
--webserver_interface=127.8.86.130
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44391
--builtin_ntp_servers=127.8.86.148:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 13:59:53.693308 10860 heartbeater.cc:347] Connected to a master server at 127.8.86.190:44391
I20240502 13:59:53.693533 10860 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:53.694049 10860 heartbeater.cc:511] Master 127.8.86.190:44391 requested a full tablet report, sending...
I20240502 13:59:53.695513 10672 ts_manager.cc:194] Registered new tserver with Master: 693080b78f6941119ae0404d7b10e687 (127.8.86.129:34875)
I20240502 13:59:53.696734 10672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.129:34855
W20240502 13:59:53.794795 10863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:53.795040 10863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:53.795114 10863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:53.798228 10863 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:53.798336 10863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:53.798460 10863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.130
I20240502 13:59:53.802054 10863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.8.86.130
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44391
--never_fsync=true
--heap_profile_path=/tmp/kudu.10863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.130
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:53.802997 10863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:53.804149 10863 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:53.812673 10868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:53.813382 10869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:53.814067 10871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:53.814903 10863 server_base.cc:1034] running on GCE node
I20240502 13:59:53.815268 10863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:53.815817 10863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:53.816963 10863 hybrid_clock.cc:648] HybridClock initialized: now 1714658393816950 us; error 29 us; skew 500 ppm
I20240502 13:59:53.818944 10863 webserver.cc:460] Webserver started at http://127.8.86.130:45397/ using document root <none> and password file <none>
I20240502 13:59:53.819475 10863 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:53.819540 10863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:53.819723 10863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:53.821297 10863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1273f709ca26429887d5f2de219eb7f5"
format_stamp: "Formatted at 2024-05-02 13:59:53 on dist-test-slave-h6v0"
I20240502 13:59:53.821763 10863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1273f709ca26429887d5f2de219eb7f5"
format_stamp: "Formatted at 2024-05-02 13:59:53 on dist-test-slave-h6v0"
I20240502 13:59:53.824952 10863 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 13:59:53.827090 10877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:53.828047 10863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:53.828159 10863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1273f709ca26429887d5f2de219eb7f5"
format_stamp: "Formatted at 2024-05-02 13:59:53 on dist-test-slave-h6v0"
I20240502 13:59:53.828239 10863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:53.851060 10863 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:53.851680 10863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:53.851838 10863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:53.852327 10863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:53.853152 10863 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:53.853200 10863 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:53.853258 10863 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:53.853289 10863 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:53.861253 10863 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.130:45649
I20240502 13:59:53.861299 10990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.130:45649 every 8 connection(s)
I20240502 13:59:53.862133 10863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 13:59:53.864112 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 10863
I20240502 13:59:53.864209 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 13:59:53.866436 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.131:0
--local_ip_for_outbound_sockets=127.8.86.131
--webserver_interface=127.8.86.131
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44391
--builtin_ntp_servers=127.8.86.148:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 13:59:53.871270 10991 heartbeater.cc:347] Connected to a master server at 127.8.86.190:44391
I20240502 13:59:53.871490 10991 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:53.872013 10991 heartbeater.cc:511] Master 127.8.86.190:44391 requested a full tablet report, sending...
I20240502 13:59:53.873019 10672 ts_manager.cc:194] Registered new tserver with Master: 1273f709ca26429887d5f2de219eb7f5 (127.8.86.130:45649)
I20240502 13:59:53.873560 10672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.130:39569
W20240502 13:59:53.970854 10994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:53.971094 10994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:53.971158 10994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:53.974058 10994 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:53.974136 10994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:53.974227 10994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.131
I20240502 13:59:53.978024 10994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.8.86.131
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44391
--never_fsync=true
--heap_profile_path=/tmp/kudu.10994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.131
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:53.978966 10994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:53.980031 10994 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:53.988548 10999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:53.989099 11000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:53.989904 11002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:53.990608 10994 server_base.cc:1034] running on GCE node
I20240502 13:59:53.991022 10994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:53.991590 10994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:53.992738 10994 hybrid_clock.cc:648] HybridClock initialized: now 1714658393992720 us; error 38 us; skew 500 ppm
I20240502 13:59:53.994647 10994 webserver.cc:460] Webserver started at http://127.8.86.131:45241/ using document root <none> and password file <none>
I20240502 13:59:53.995183 10994 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:53.995252 10994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:53.995441 10994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:53.997088 10994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0161477bb63641eba29a492db0d9bddb"
format_stamp: "Formatted at 2024-05-02 13:59:53 on dist-test-slave-h6v0"
I20240502 13:59:53.997578 10994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0161477bb63641eba29a492db0d9bddb"
format_stamp: "Formatted at 2024-05-02 13:59:53 on dist-test-slave-h6v0"
I20240502 13:59:54.000883 10994 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240502 13:59:54.003129 11008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:54.004127 10994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 13:59:54.004243 10994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0161477bb63641eba29a492db0d9bddb"
format_stamp: "Formatted at 2024-05-02 13:59:53 on dist-test-slave-h6v0"
I20240502 13:59:54.004328 10994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:54.053795 10994 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:54.054428 10994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:54.054590 10994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:54.055074 10994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:54.055929 10994 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:54.055977 10994 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:54.056037 10994 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:54.056062 10994 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:54.064388 10994 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.131:38209
I20240502 13:59:54.064455 11121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.131:38209 every 8 connection(s)
I20240502 13:59:54.065399 10994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 13:59:54.072993 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 10994
I20240502 13:59:54.073107 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 13:59:54.074110 11122 heartbeater.cc:347] Connected to a master server at 127.8.86.190:44391
I20240502 13:59:54.074340 11122 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:54.074839 11122 heartbeater.cc:511] Master 127.8.86.190:44391 requested a full tablet report, sending...
I20240502 13:59:54.074970 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.132:0
--local_ip_for_outbound_sockets=127.8.86.132
--webserver_interface=127.8.86.132
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44391
--builtin_ntp_servers=127.8.86.148:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 13:59:54.075783 10672 ts_manager.cc:194] Registered new tserver with Master: 0161477bb63641eba29a492db0d9bddb (127.8.86.131:38209)
I20240502 13:59:54.076313 10672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.131:39885
W20240502 13:59:54.181489 11125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:54.181725 11125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:54.181803 11125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:54.184827 11125 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:54.184903 11125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:54.185000 11125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.132
I20240502 13:59:54.188602 11125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.86.132
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44391
--never_fsync=true
--heap_profile_path=/tmp/kudu.11125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.132
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:54.189488 11125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:54.190588 11125 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:54.199445 11131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:54.200353 11130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:54.201202 11133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:54.201882 11125 server_base.cc:1034] running on GCE node
I20240502 13:59:54.202266 11125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:54.202910 11125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:54.204134 11125 hybrid_clock.cc:648] HybridClock initialized: now 1714658394204117 us; error 27 us; skew 500 ppm
I20240502 13:59:54.206187 11125 webserver.cc:460] Webserver started at http://127.8.86.132:44603/ using document root <none> and password file <none>
I20240502 13:59:54.206789 11125 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:54.206849 11125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:54.207047 11125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:54.208794 11125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b44cd86544ba43c9af2dc69daa73ba19"
format_stamp: "Formatted at 2024-05-02 13:59:54 on dist-test-slave-h6v0"
I20240502 13:59:54.209271 11125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b44cd86544ba43c9af2dc69daa73ba19"
format_stamp: "Formatted at 2024-05-02 13:59:54 on dist-test-slave-h6v0"
I20240502 13:59:54.212769 11125 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 13:59:54.214907 11139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:54.215848 11125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:54.215945 11125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b44cd86544ba43c9af2dc69daa73ba19"
format_stamp: "Formatted at 2024-05-02 13:59:54 on dist-test-slave-h6v0"
I20240502 13:59:54.216035 11125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:54.236142 11125 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:54.236747 11125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:54.236934 11125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:54.237460 11125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:54.238392 11125 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:54.238438 11125 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:54.238484 11125 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:54.238516 11125 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:54.246723 11125 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.132:46773
I20240502 13:59:54.246768 11252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.132:46773 every 8 connection(s)
I20240502 13:59:54.247628 11125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 13:59:54.251582 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 11125
I20240502 13:59:54.251680 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 13:59:54.253644 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.133:0
--local_ip_for_outbound_sockets=127.8.86.133
--webserver_interface=127.8.86.133
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44391
--builtin_ntp_servers=127.8.86.148:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 13:59:54.256204 11253 heartbeater.cc:347] Connected to a master server at 127.8.86.190:44391
I20240502 13:59:54.256409 11253 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:54.256937 11253 heartbeater.cc:511] Master 127.8.86.190:44391 requested a full tablet report, sending...
I20240502 13:59:54.257887 10672 ts_manager.cc:194] Registered new tserver with Master: b44cd86544ba43c9af2dc69daa73ba19 (127.8.86.132:46773)
I20240502 13:59:54.258484 10672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.132:40527
W20240502 13:59:54.359436 11256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:54.359664 11256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:54.359737 11256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:54.362658 11256 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:54.362728 11256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:54.362813 11256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.133
I20240502 13:59:54.366367 11256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.86.133
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44391
--never_fsync=true
--heap_profile_path=/tmp/kudu.11256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.133
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:54.367265 11256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:54.368355 11256 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:54.376770 11261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:54.377604 11264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:54.378330 11262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:54.379104 11256 server_base.cc:1034] running on GCE node
I20240502 13:59:54.379438 11256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:54.379923 11256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:54.381083 11256 hybrid_clock.cc:648] HybridClock initialized: now 1714658394381068 us; error 27 us; skew 500 ppm
I20240502 13:59:54.382987 11256 webserver.cc:460] Webserver started at http://127.8.86.133:36397/ using document root <none> and password file <none>
I20240502 13:59:54.383496 11256 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:54.383551 11256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:54.383713 11256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:54.385258 11256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "f3c65f4a512c4fdaa88831af9301495a"
format_stamp: "Formatted at 2024-05-02 13:59:54 on dist-test-slave-h6v0"
I20240502 13:59:54.385748 11256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "f3c65f4a512c4fdaa88831af9301495a"
format_stamp: "Formatted at 2024-05-02 13:59:54 on dist-test-slave-h6v0"
I20240502 13:59:54.388900 11256 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 13:59:54.390924 11270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:54.391858 11256 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:54.391955 11256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f3c65f4a512c4fdaa88831af9301495a"
format_stamp: "Formatted at 2024-05-02 13:59:54 on dist-test-slave-h6v0"
I20240502 13:59:54.392055 11256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:54.408591 11256 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:54.409158 11256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:54.409314 11256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:54.409806 11256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:54.410679 11256 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:54.410729 11256 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:54.410794 11256 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:54.410820 11256 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:54.418702 11256 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.133:36429
I20240502 13:59:54.418758 11383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.133:36429 every 8 connection(s)
I20240502 13:59:54.419620 11256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 13:59:54.427963 11384 heartbeater.cc:347] Connected to a master server at 127.8.86.190:44391
I20240502 13:59:54.428158 11384 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:54.428638 11384 heartbeater.cc:511] Master 127.8.86.190:44391 requested a full tablet report, sending...
I20240502 13:59:54.429330 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 11256
I20240502 13:59:54.429427 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 13:59:54.429556 10672 ts_manager.cc:194] Registered new tserver with Master: f3c65f4a512c4fdaa88831af9301495a (127.8.86.133:36429)
I20240502 13:59:54.430107 10672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.133:41939
I20240502 13:59:54.431542 8538 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 13:59:54.440148 10672 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56908:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 13:59:54.459050 11056 tablet_service.cc:1467] Processing CreateTablet for tablet 9f37cc99c7f449e7bf7a91438eea77c4 (DEFAULT_TABLE table=TestTable [id=54c22d253ae442ce962673af58cdf40b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:54.459050 11187 tablet_service.cc:1467] Processing CreateTablet for tablet 9f37cc99c7f449e7bf7a91438eea77c4 (DEFAULT_TABLE table=TestTable [id=54c22d253ae442ce962673af58cdf40b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:54.459050 10925 tablet_service.cc:1467] Processing CreateTablet for tablet 9f37cc99c7f449e7bf7a91438eea77c4 (DEFAULT_TABLE table=TestTable [id=54c22d253ae442ce962673af58cdf40b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:54.460023 11187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f37cc99c7f449e7bf7a91438eea77c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:54.460022 10925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f37cc99c7f449e7bf7a91438eea77c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:54.460023 11056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f37cc99c7f449e7bf7a91438eea77c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:54.466750 11401 tablet_bootstrap.cc:492] T 9f37cc99c7f449e7bf7a91438eea77c4 P 0161477bb63641eba29a492db0d9bddb: Bootstrap starting.
I20240502 13:59:54.466938 11402 tablet_bootstrap.cc:492] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5: Bootstrap starting.
I20240502 13:59:54.467738 11403 tablet_bootstrap.cc:492] T 9f37cc99c7f449e7bf7a91438eea77c4 P b44cd86544ba43c9af2dc69daa73ba19: Bootstrap starting.
I20240502 13:59:54.468968 11401 tablet_bootstrap.cc:654] T 9f37cc99c7f449e7bf7a91438eea77c4 P 0161477bb63641eba29a492db0d9bddb: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:54.468971 11402 tablet_bootstrap.cc:654] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:54.469735 11401 log.cc:824] T 9f37cc99c7f449e7bf7a91438eea77c4 P 0161477bb63641eba29a492db0d9bddb: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:54.469744 11402 log.cc:824] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:54.470537 11403 tablet_bootstrap.cc:654] T 9f37cc99c7f449e7bf7a91438eea77c4 P b44cd86544ba43c9af2dc69daa73ba19: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:54.471257 11403 log.cc:824] T 9f37cc99c7f449e7bf7a91438eea77c4 P b44cd86544ba43c9af2dc69daa73ba19: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:54.471405 11402 tablet_bootstrap.cc:492] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5: No bootstrap required, opened a new log
I20240502 13:59:54.471438 11401 tablet_bootstrap.cc:492] T 9f37cc99c7f449e7bf7a91438eea77c4 P 0161477bb63641eba29a492db0d9bddb: No bootstrap required, opened a new log
I20240502 13:59:54.471576 11402 ts_tablet_manager.cc:1367] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 13:59:54.471585 11401 ts_tablet_manager.cc:1367] T 9f37cc99c7f449e7bf7a91438eea77c4 P 0161477bb63641eba29a492db0d9bddb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 13:59:54.472864 11403 tablet_bootstrap.cc:492] T 9f37cc99c7f449e7bf7a91438eea77c4 P b44cd86544ba43c9af2dc69daa73ba19: No bootstrap required, opened a new log
I20240502 13:59:54.473173 11403 ts_tablet_manager.cc:1367] T 9f37cc99c7f449e7bf7a91438eea77c4 P b44cd86544ba43c9af2dc69daa73ba19: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 13:59:54.474710 11401 raft_consensus.cc:348] T 9f37cc99c7f449e7bf7a91438eea77c4 P 0161477bb63641eba29a492db0d9bddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273f709ca26429887d5f2de219eb7f5" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 45649 } } peers { permanent_uuid: "b44cd86544ba43c9af2dc69daa73ba19" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 46773 } } peers { permanent_uuid: "0161477bb63641eba29a492db0d9bddb" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 38209 } }
I20240502 13:59:54.474879 11401 raft_consensus.cc:374] T 9f37cc99c7f449e7bf7a91438eea77c4 P 0161477bb63641eba29a492db0d9bddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:54.474942 11401 raft_consensus.cc:729] T 9f37cc99c7f449e7bf7a91438eea77c4 P 0161477bb63641eba29a492db0d9bddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0161477bb63641eba29a492db0d9bddb, State: Initialized, Role: FOLLOWER
I20240502 13:59:54.475039 11402 raft_consensus.cc:348] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273f709ca26429887d5f2de219eb7f5" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 45649 } } peers { permanent_uuid: "b44cd86544ba43c9af2dc69daa73ba19" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 46773 } } peers { permanent_uuid: "0161477bb63641eba29a492db0d9bddb" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 38209 } }
I20240502 13:59:54.475289 11402 raft_consensus.cc:374] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:54.475392 11402 raft_consensus.cc:729] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1273f709ca26429887d5f2de219eb7f5, State: Initialized, Role: FOLLOWER
I20240502 13:59:54.475389 11401 consensus_queue.cc:260] T 9f37cc99c7f449e7bf7a91438eea77c4 P 0161477bb63641eba29a492db0d9bddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273f709ca26429887d5f2de219eb7f5" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 45649 } } peers { permanent_uuid: "b44cd86544ba43c9af2dc69daa73ba19" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 46773 } } peers { permanent_uuid: "0161477bb63641eba29a492db0d9bddb" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 38209 } }
I20240502 13:59:54.475818 11402 consensus_queue.cc:260] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273f709ca26429887d5f2de219eb7f5" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 45649 } } peers { permanent_uuid: "b44cd86544ba43c9af2dc69daa73ba19" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 46773 } } peers { permanent_uuid: "0161477bb63641eba29a492db0d9bddb" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 38209 } }
I20240502 13:59:54.476150 11401 ts_tablet_manager.cc:1398] T 9f37cc99c7f449e7bf7a91438eea77c4 P 0161477bb63641eba29a492db0d9bddb: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 13:59:54.476547 10991 heartbeater.cc:503] Master 127.8.86.190:44391 was elected leader, sending a full tablet report...
I20240502 13:59:54.476547 11122 heartbeater.cc:503] Master 127.8.86.190:44391 was elected leader, sending a full tablet report...
I20240502 13:59:54.476704 11402 ts_tablet_manager.cc:1398] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20240502 13:59:54.477577 11403 raft_consensus.cc:348] T 9f37cc99c7f449e7bf7a91438eea77c4 P b44cd86544ba43c9af2dc69daa73ba19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273f709ca26429887d5f2de219eb7f5" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 45649 } } peers { permanent_uuid: "b44cd86544ba43c9af2dc69daa73ba19" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 46773 } } peers { permanent_uuid: "0161477bb63641eba29a492db0d9bddb" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 38209 } }
I20240502 13:59:54.477787 11403 raft_consensus.cc:374] T 9f37cc99c7f449e7bf7a91438eea77c4 P b44cd86544ba43c9af2dc69daa73ba19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:54.477852 11403 raft_consensus.cc:729] T 9f37cc99c7f449e7bf7a91438eea77c4 P b44cd86544ba43c9af2dc69daa73ba19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44cd86544ba43c9af2dc69daa73ba19, State: Initialized, Role: FOLLOWER
I20240502 13:59:54.478214 11403 consensus_queue.cc:260] T 9f37cc99c7f449e7bf7a91438eea77c4 P b44cd86544ba43c9af2dc69daa73ba19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273f709ca26429887d5f2de219eb7f5" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 45649 } } peers { permanent_uuid: "b44cd86544ba43c9af2dc69daa73ba19" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 46773 } } peers { permanent_uuid: "0161477bb63641eba29a492db0d9bddb" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 38209 } }
I20240502 13:59:54.479019 11403 ts_tablet_manager.cc:1398] T 9f37cc99c7f449e7bf7a91438eea77c4 P b44cd86544ba43c9af2dc69daa73ba19: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 13:59:54.479171 11253 heartbeater.cc:503] Master 127.8.86.190:44391 was elected leader, sending a full tablet report...
I20240502 13:59:54.496984 11408 raft_consensus.cc:482] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:54.497157 11408 raft_consensus.cc:504] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273f709ca26429887d5f2de219eb7f5" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 45649 } } peers { permanent_uuid: "b44cd86544ba43c9af2dc69daa73ba19" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 46773 } } peers { permanent_uuid: "0161477bb63641eba29a492db0d9bddb" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 38209 } }
I20240502 13:59:54.498157 11408 leader_election.cc:288] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b44cd86544ba43c9af2dc69daa73ba19 (127.8.86.132:46773), 0161477bb63641eba29a492db0d9bddb (127.8.86.131:38209)
W20240502 13:59:54.498481 11254 tablet.cc:2369] T 9f37cc99c7f449e7bf7a91438eea77c4 P b44cd86544ba43c9af2dc69daa73ba19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 13:59:54.501910 11076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f37cc99c7f449e7bf7a91438eea77c4" candidate_uuid: "1273f709ca26429887d5f2de219eb7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0161477bb63641eba29a492db0d9bddb" is_pre_election: true
I20240502 13:59:54.501965 11207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f37cc99c7f449e7bf7a91438eea77c4" candidate_uuid: "1273f709ca26429887d5f2de219eb7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44cd86544ba43c9af2dc69daa73ba19" is_pre_election: true
I20240502 13:59:54.502148 11076 raft_consensus.cc:2454] T 9f37cc99c7f449e7bf7a91438eea77c4 P 0161477bb63641eba29a492db0d9bddb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1273f709ca26429887d5f2de219eb7f5 in term 0.
I20240502 13:59:54.502182 11207 raft_consensus.cc:2454] T 9f37cc99c7f449e7bf7a91438eea77c4 P b44cd86544ba43c9af2dc69daa73ba19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1273f709ca26429887d5f2de219eb7f5 in term 0.
I20240502 13:59:54.502547 10881 leader_election.cc:302] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0161477bb63641eba29a492db0d9bddb, 1273f709ca26429887d5f2de219eb7f5; no voters:
I20240502 13:59:54.502758 11408 raft_consensus.cc:2781] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:54.502822 11408 raft_consensus.cc:482] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:54.502849 11408 raft_consensus.cc:3037] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:54.503669 11408 raft_consensus.cc:504] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273f709ca26429887d5f2de219eb7f5" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 45649 } } peers { permanent_uuid: "b44cd86544ba43c9af2dc69daa73ba19" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 46773 } } peers { permanent_uuid: "0161477bb63641eba29a492db0d9bddb" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 38209 } }
I20240502 13:59:54.503968 11408 leader_election.cc:288] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 [CANDIDATE]: Term 1 election: Requested vote from peers b44cd86544ba43c9af2dc69daa73ba19 (127.8.86.132:46773), 0161477bb63641eba29a492db0d9bddb (127.8.86.131:38209)
I20240502 13:59:54.504247 11207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f37cc99c7f449e7bf7a91438eea77c4" candidate_uuid: "1273f709ca26429887d5f2de219eb7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44cd86544ba43c9af2dc69daa73ba19"
I20240502 13:59:54.504304 11076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f37cc99c7f449e7bf7a91438eea77c4" candidate_uuid: "1273f709ca26429887d5f2de219eb7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0161477bb63641eba29a492db0d9bddb"
I20240502 13:59:54.504349 11207 raft_consensus.cc:3037] T 9f37cc99c7f449e7bf7a91438eea77c4 P b44cd86544ba43c9af2dc69daa73ba19 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:54.504423 11076 raft_consensus.cc:3037] T 9f37cc99c7f449e7bf7a91438eea77c4 P 0161477bb63641eba29a492db0d9bddb [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:54.505349 11207 raft_consensus.cc:2454] T 9f37cc99c7f449e7bf7a91438eea77c4 P b44cd86544ba43c9af2dc69daa73ba19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1273f709ca26429887d5f2de219eb7f5 in term 1.
I20240502 13:59:54.505442 11076 raft_consensus.cc:2454] T 9f37cc99c7f449e7bf7a91438eea77c4 P 0161477bb63641eba29a492db0d9bddb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1273f709ca26429887d5f2de219eb7f5 in term 1.
I20240502 13:59:54.505690 10881 leader_election.cc:302] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1273f709ca26429887d5f2de219eb7f5, b44cd86544ba43c9af2dc69daa73ba19; no voters:
I20240502 13:59:54.505856 11408 raft_consensus.cc:2781] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:54.506033 11408 raft_consensus.cc:686] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 [term 1 LEADER]: Becoming Leader. State: Replica: 1273f709ca26429887d5f2de219eb7f5, State: Running, Role: LEADER
I20240502 13:59:54.506354 11408 consensus_queue.cc:237] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273f709ca26429887d5f2de219eb7f5" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 45649 } } peers { permanent_uuid: "b44cd86544ba43c9af2dc69daa73ba19" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 46773 } } peers { permanent_uuid: "0161477bb63641eba29a492db0d9bddb" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 38209 } }
I20240502 13:59:54.510789 10672 catalog_manager.cc:5484] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1273f709ca26429887d5f2de219eb7f5 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "1273f709ca26429887d5f2de219eb7f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1273f709ca26429887d5f2de219eb7f5" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 45649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b44cd86544ba43c9af2dc69daa73ba19" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 46773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0161477bb63641eba29a492db0d9bddb" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 38209 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:54.520838 8538 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 13:59:54.522204 8538 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 693080b78f6941119ae0404d7b10e687 to finish bootstrapping
I20240502 13:59:54.527686 8538 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1273f709ca26429887d5f2de219eb7f5 to finish bootstrapping
I20240502 13:59:54.531787 8538 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0161477bb63641eba29a492db0d9bddb to finish bootstrapping
I20240502 13:59:54.536074 8538 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b44cd86544ba43c9af2dc69daa73ba19 to finish bootstrapping
I20240502 13:59:54.540174 8538 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f3c65f4a512c4fdaa88831af9301495a to finish bootstrapping
W20240502 13:59:54.566569 11123 tablet.cc:2369] T 9f37cc99c7f449e7bf7a91438eea77c4 P 0161477bb63641eba29a492db0d9bddb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 13:59:54.613356 10992 tablet.cc:2369] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 13:59:54.698974 10860 heartbeater.cc:503] Master 127.8.86.190:44391 was elected leader, sending a full tablet report...
I20240502 13:59:54.720094 10925 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:54.721271 11056 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:54.721840 10794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:54.722416 11187 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:54.723609 11318 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:54.810164 11056 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:54.810935 10794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:54.811127 10925 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:54.812636 11318 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:54.830256 11187 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:54.998747 11408 consensus_queue.cc:1035] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0161477bb63641eba29a492db0d9bddb" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:55.005425 11408 consensus_queue.cc:1035] T 9f37cc99c7f449e7bf7a91438eea77c4 P 1273f709ca26429887d5f2de219eb7f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b44cd86544ba43c9af2dc69daa73ba19" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 46773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:55.031100 11056 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:55.031344 11187 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:55.031689 10794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:55.031837 11318 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:55.032634 10925 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:55.123180 11056 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:55.124850 10925 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:55.125243 11187 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:55.126005 10794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:55.133960 11318 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:55.217952 10925 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:55.220129 11318 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:55.224326 11187 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:55.227054 11056 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:55.235801 10794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:55.264791 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 10732
I20240502 13:59:55.270861 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 10863
I20240502 13:59:55.276814 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 10994
I20240502 13:59:55.282521 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 11125
I20240502 13:59:55.288444 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 11256
I20240502 13:59:55.293977 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 10641
2024-05-02T13:59:55Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (2020 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2020 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/3
I20240502 13:59:55.316762 8538 test_util.cc:268] Using random seed: 76571157
I20240502 13:59:55.317509 8538 ts_itest-base.cc:112] Starting cluster with:
I20240502 13:59:55.317544 8538 ts_itest-base.cc:113] --------------
I20240502 13:59:55.317555 8538 ts_itest-base.cc:114] 7 tablet servers
I20240502 13:59:55.317566 8538 ts_itest-base.cc:115] 2 replicas per TS
I20240502 13:59:55.317579 8538 ts_itest-base.cc:116] --------------
2024-05-02T13:59:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T13:59:55Z Disabled control of system clock
I20240502 13:59:55.325176 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.86.190:44201
--webserver_interface=127.8.86.190
--webserver_port=0
--builtin_ntp_servers=127.8.86.148:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.86.190:44201
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 13:59:55.426143 11595 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 13:59:55.426419 11595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:55.426448 11595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:55.426497 11595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:55.429497 11595 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:55.429558 11595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 13:59:55.429580 11595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:55.429605 11595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 13:59:55.429621 11595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 13:59:55.433249 11595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.8.86.190:44201
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.86.190:44201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.8.86.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.11595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:55.434059 11595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:55.435148 11595 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:55.442657 11600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:55.443442 11601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:55.443991 11603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:55.444747 11595 server_base.cc:1034] running on GCE node
I20240502 13:59:55.445166 11595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:55.446079 11595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:55.447234 11595 hybrid_clock.cc:648] HybridClock initialized: now 1714658395447212 us; error 36 us; skew 500 ppm
I20240502 13:59:55.449136 11595 webserver.cc:460] Webserver started at http://127.8.86.190:43429/ using document root <none> and password file <none>
I20240502 13:59:55.449664 11595 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:55.449719 11595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:55.449915 11595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:55.451586 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "12453ce20bf34d93b25ce6a40cfe613c"
format_stamp: "Formatted at 2024-05-02 13:59:55 on dist-test-slave-h6v0"
I20240502 13:59:55.452046 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "12453ce20bf34d93b25ce6a40cfe613c"
format_stamp: "Formatted at 2024-05-02 13:59:55 on dist-test-slave-h6v0"
I20240502 13:59:55.455163 11595 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 13:59:55.457235 11609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:55.458205 11595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:55.458341 11595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
uuid: "12453ce20bf34d93b25ce6a40cfe613c"
format_stamp: "Formatted at 2024-05-02 13:59:55 on dist-test-slave-h6v0"
I20240502 13:59:55.458426 11595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:55.468672 11595 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:55.469251 11595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:55.469409 11595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:55.475795 11595 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.190:44201
I20240502 13:59:55.475802 11661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.190:44201 every 8 connection(s)
I20240502 13:59:55.476686 11595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 13:59:55.479183 11662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:55.479558 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 11595
I20240502 13:59:55.479641 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 13:59:55.484440 11662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c: Bootstrap starting.
I20240502 13:59:55.486698 11662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:55.487376 11662 log.cc:824] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:55.488978 11662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c: No bootstrap required, opened a new log
I20240502 13:59:55.491928 11662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12453ce20bf34d93b25ce6a40cfe613c" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 44201 } }
I20240502 13:59:55.492115 11662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:55.492187 11662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12453ce20bf34d93b25ce6a40cfe613c, State: Initialized, Role: FOLLOWER
I20240502 13:59:55.492622 11662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12453ce20bf34d93b25ce6a40cfe613c" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 44201 } }
I20240502 13:59:55.492767 11662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:55.492830 11662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:55.492928 11662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:55.493643 11662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12453ce20bf34d93b25ce6a40cfe613c" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 44201 } }
I20240502 13:59:55.493995 11662 leader_election.cc:302] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12453ce20bf34d93b25ce6a40cfe613c; no voters:
I20240502 13:59:55.494307 11662 leader_election.cc:288] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:55.494722 11667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:55.494917 11667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c [term 1 LEADER]: Becoming Leader. State: Replica: 12453ce20bf34d93b25ce6a40cfe613c, State: Running, Role: LEADER
I20240502 13:59:55.495187 11667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12453ce20bf34d93b25ce6a40cfe613c" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 44201 } }
I20240502 13:59:55.495409 11662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c [sys.catalog]: configured and running, proceeding with master startup.
I20240502 13:59:55.496627 11669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12453ce20bf34d93b25ce6a40cfe613c. Latest consensus state: current_term: 1 leader_uuid: "12453ce20bf34d93b25ce6a40cfe613c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12453ce20bf34d93b25ce6a40cfe613c" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 44201 } } }
I20240502 13:59:55.496637 11668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12453ce20bf34d93b25ce6a40cfe613c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12453ce20bf34d93b25ce6a40cfe613c" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 44201 } } }
I20240502 13:59:55.496881 11668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:55.497444 11669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:55.497658 11677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 13:59:55.500099 11677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 13:59:55.504971 11677 catalog_manager.cc:1347] Generated new cluster ID: 57599e6a0fa049cbbb0cd3d5ca840498
I20240502 13:59:55.505038 11677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 13:59:55.522114 11677 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 13:59:55.523288 11677 catalog_manager.cc:1504] Loading token signing keys...
I20240502 13:59:55.532198 11677 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 12453ce20bf34d93b25ce6a40cfe613c: Generated new TSK 0
I20240502 13:59:55.532971 11677 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 13:59:55.537184 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.129:0
--local_ip_for_outbound_sockets=127.8.86.129
--webserver_interface=127.8.86.129
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44201
--builtin_ntp_servers=127.8.86.148:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 13:59:55.641896 11686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:55.642086 11686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:55.642134 11686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:55.644965 11686 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:55.645041 11686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:55.645114 11686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.129
I20240502 13:59:55.648766 11686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.8.86.129
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44201
--never_fsync=true
--heap_profile_path=/tmp/kudu.11686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.129
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:55.649631 11686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:55.650768 11686 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:55.658735 11694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:55.659202 11692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:55.659998 11691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:55.660768 11686 server_base.cc:1034] running on GCE node
I20240502 13:59:55.661068 11686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:55.661586 11686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:55.662756 11686 hybrid_clock.cc:648] HybridClock initialized: now 1714658395662740 us; error 36 us; skew 500 ppm
I20240502 13:59:55.667827 11686 webserver.cc:460] Webserver started at http://127.8.86.129:39603/ using document root <none> and password file <none>
I20240502 13:59:55.668396 11686 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:55.668501 11686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:55.668766 11686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:55.670434 11686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
format_stamp: "Formatted at 2024-05-02 13:59:55 on dist-test-slave-h6v0"
I20240502 13:59:55.670876 11686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
format_stamp: "Formatted at 2024-05-02 13:59:55 on dist-test-slave-h6v0"
I20240502 13:59:55.673976 11686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 13:59:55.676034 11700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:55.676927 11686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:55.677047 11686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
format_stamp: "Formatted at 2024-05-02 13:59:55 on dist-test-slave-h6v0"
I20240502 13:59:55.677127 11686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:55.691358 11686 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:55.691933 11686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:55.692085 11686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:55.692580 11686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:55.693395 11686 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:55.693442 11686 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:55.693495 11686 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:55.693526 11686 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:55.702124 11686 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.129:39411
I20240502 13:59:55.702153 11813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.129:39411 every 8 connection(s)
I20240502 13:59:55.702958 11686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 13:59:55.711366 11814 heartbeater.cc:347] Connected to a master server at 127.8.86.190:44201
I20240502 13:59:55.711580 11814 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:55.712098 11814 heartbeater.cc:511] Master 127.8.86.190:44201 requested a full tablet report, sending...
I20240502 13:59:55.713025 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 11686
I20240502 13:59:55.713120 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 13:59:55.713436 11626 ts_manager.cc:194] Registered new tserver with Master: 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 13:59:55.714726 11626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.129:46657
I20240502 13:59:55.715160 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.130:0
--local_ip_for_outbound_sockets=127.8.86.130
--webserver_interface=127.8.86.130
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44201
--builtin_ntp_servers=127.8.86.148:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 13:59:55.818940 11817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:55.819140 11817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:55.819183 11817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:55.822039 11817 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:55.822094 11817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:55.822155 11817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.130
I20240502 13:59:55.825695 11817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.8.86.130
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44201
--never_fsync=true
--heap_profile_path=/tmp/kudu.11817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.130
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:55.826512 11817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:55.827711 11817 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:55.836433 11823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:55.837198 11825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:55.837970 11817 server_base.cc:1034] running on GCE node
W20240502 13:59:55.838733 11822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:55.839110 11817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:55.839658 11817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:55.840883 11817 hybrid_clock.cc:648] HybridClock initialized: now 1714658395840857 us; error 44 us; skew 500 ppm
I20240502 13:59:55.842900 11817 webserver.cc:460] Webserver started at http://127.8.86.130:34169/ using document root <none> and password file <none>
I20240502 13:59:55.843447 11817 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:55.843513 11817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:55.843717 11817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:55.845328 11817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "cbf92f56d7144a22a6d482c73c7c7ea9"
format_stamp: "Formatted at 2024-05-02 13:59:55 on dist-test-slave-h6v0"
I20240502 13:59:55.845863 11817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "cbf92f56d7144a22a6d482c73c7c7ea9"
format_stamp: "Formatted at 2024-05-02 13:59:55 on dist-test-slave-h6v0"
I20240502 13:59:55.849164 11817 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240502 13:59:55.851336 11831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:55.852355 11817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:55.852471 11817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "cbf92f56d7144a22a6d482c73c7c7ea9"
format_stamp: "Formatted at 2024-05-02 13:59:55 on dist-test-slave-h6v0"
I20240502 13:59:55.852564 11817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:55.864416 11817 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:55.865007 11817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:55.865162 11817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:55.865674 11817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:55.866561 11817 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:55.866618 11817 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:55.866675 11817 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:55.866703 11817 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:55.875031 11817 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.130:44377
I20240502 13:59:55.875061 11944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.130:44377 every 8 connection(s)
I20240502 13:59:55.875912 11817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 13:59:55.880942 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 11817
I20240502 13:59:55.881021 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 13:59:55.883121 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.131:0
--local_ip_for_outbound_sockets=127.8.86.131
--webserver_interface=127.8.86.131
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44201
--builtin_ntp_servers=127.8.86.148:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 13:59:55.885171 11945 heartbeater.cc:347] Connected to a master server at 127.8.86.190:44201
I20240502 13:59:55.885385 11945 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:55.885893 11945 heartbeater.cc:511] Master 127.8.86.190:44201 requested a full tablet report, sending...
I20240502 13:59:55.886904 11626 ts_manager.cc:194] Registered new tserver with Master: cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377)
I20240502 13:59:55.887444 11626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.130:46291
W20240502 13:59:55.985114 11948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:55.985347 11948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:55.985427 11948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:55.988729 11948 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:55.988804 11948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:55.988894 11948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.131
I20240502 13:59:55.992301 11948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.8.86.131
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44201
--never_fsync=true
--heap_profile_path=/tmp/kudu.11948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.131
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:55.993130 11948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:55.994207 11948 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:56.002347 11956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:56.003018 11954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:56.003790 11953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:56.004473 11948 server_base.cc:1034] running on GCE node
I20240502 13:59:56.004808 11948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:56.005296 11948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:56.006438 11948 hybrid_clock.cc:648] HybridClock initialized: now 1714658396006427 us; error 22 us; skew 500 ppm
I20240502 13:59:56.008253 11948 webserver.cc:460] Webserver started at http://127.8.86.131:39765/ using document root <none> and password file <none>
I20240502 13:59:56.008760 11948 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:56.008834 11948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:56.008987 11948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:56.010500 11948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "65a078666d30488f9c204d93e6394ef9"
format_stamp: "Formatted at 2024-05-02 13:59:56 on dist-test-slave-h6v0"
I20240502 13:59:56.010919 11948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "65a078666d30488f9c204d93e6394ef9"
format_stamp: "Formatted at 2024-05-02 13:59:56 on dist-test-slave-h6v0"
I20240502 13:59:56.013861 11948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 13:59:56.015841 11962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:56.016741 11948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:56.016832 11948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "65a078666d30488f9c204d93e6394ef9"
format_stamp: "Formatted at 2024-05-02 13:59:56 on dist-test-slave-h6v0"
I20240502 13:59:56.016901 11948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:56.035776 11948 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:56.036360 11948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:56.036482 11948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:56.036924 11948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:56.037765 11948 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:56.037811 11948 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:56.037849 11948 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:56.037870 11948 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:56.045696 11948 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.131:45325
I20240502 13:59:56.045725 12075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.131:45325 every 8 connection(s)
I20240502 13:59:56.046612 11948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 13:59:56.048632 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 11948
I20240502 13:59:56.048740 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 13:59:56.050807 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.132:0
--local_ip_for_outbound_sockets=127.8.86.132
--webserver_interface=127.8.86.132
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44201
--builtin_ntp_servers=127.8.86.148:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 13:59:56.055419 12076 heartbeater.cc:347] Connected to a master server at 127.8.86.190:44201
I20240502 13:59:56.055671 12076 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:56.056340 12076 heartbeater.cc:511] Master 127.8.86.190:44201 requested a full tablet report, sending...
I20240502 13:59:56.057536 11626 ts_manager.cc:194] Registered new tserver with Master: 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 13:59:56.058073 11626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.131:41239
W20240502 13:59:56.154790 12079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:56.155000 12079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:56.155043 12079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:56.158033 12079 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:56.158102 12079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:56.158207 12079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.132
I20240502 13:59:56.161772 12079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.86.132
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44201
--never_fsync=true
--heap_profile_path=/tmp/kudu.12079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.132
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:56.162655 12079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:56.163746 12079 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:56.172142 12084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:56.172658 12087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:56.173575 12085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:56.174242 12079 server_base.cc:1034] running on GCE node
I20240502 13:59:56.174679 12079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:56.175227 12079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:56.176409 12079 hybrid_clock.cc:648] HybridClock initialized: now 1714658396176388 us; error 32 us; skew 500 ppm
I20240502 13:59:56.178354 12079 webserver.cc:460] Webserver started at http://127.8.86.132:45443/ using document root <none> and password file <none>
I20240502 13:59:56.178871 12079 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:56.178929 12079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:56.179090 12079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:56.180707 12079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4f6f2d2b1569415cbbbea2466dc28e28"
format_stamp: "Formatted at 2024-05-02 13:59:56 on dist-test-slave-h6v0"
I20240502 13:59:56.181198 12079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4f6f2d2b1569415cbbbea2466dc28e28"
format_stamp: "Formatted at 2024-05-02 13:59:56 on dist-test-slave-h6v0"
I20240502 13:59:56.184449 12079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 13:59:56.186614 12093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:56.187639 12079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 13:59:56.187754 12079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4f6f2d2b1569415cbbbea2466dc28e28"
format_stamp: "Formatted at 2024-05-02 13:59:56 on dist-test-slave-h6v0"
I20240502 13:59:56.187849 12079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:56.206777 12079 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:56.207407 12079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:56.207561 12079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:56.208091 12079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:56.209014 12079 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:56.209064 12079 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:56.209120 12079 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:56.209152 12079 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:56.217160 12079 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.132:42709
I20240502 13:59:56.217202 12206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.132:42709 every 8 connection(s)
I20240502 13:59:56.218111 12079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 13:59:56.226357 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 12079
I20240502 13:59:56.226467 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 13:59:56.226738 12207 heartbeater.cc:347] Connected to a master server at 127.8.86.190:44201
I20240502 13:59:56.226953 12207 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:56.227578 12207 heartbeater.cc:511] Master 127.8.86.190:44201 requested a full tablet report, sending...
I20240502 13:59:56.228572 11626 ts_manager.cc:194] Registered new tserver with Master: 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132:42709)
I20240502 13:59:56.228627 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.133:0
--local_ip_for_outbound_sockets=127.8.86.133
--webserver_interface=127.8.86.133
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44201
--builtin_ntp_servers=127.8.86.148:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 13:59:56.229121 11626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.132:43639
W20240502 13:59:56.332710 12210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:56.332926 12210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:56.332970 12210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:56.335942 12210 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:56.336009 12210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:56.336094 12210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.133
I20240502 13:59:56.339637 12210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.86.133
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44201
--never_fsync=true
--heap_profile_path=/tmp/kudu.12210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.133
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:56.340509 12210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:56.341612 12210 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:56.349979 12215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:56.350751 12218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:56.351398 12216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:56.352200 12210 server_base.cc:1034] running on GCE node
I20240502 13:59:56.352581 12210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:56.353106 12210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:56.354261 12210 hybrid_clock.cc:648] HybridClock initialized: now 1714658396354244 us; error 36 us; skew 500 ppm
I20240502 13:59:56.356117 12210 webserver.cc:460] Webserver started at http://127.8.86.133:36609/ using document root <none> and password file <none>
I20240502 13:59:56.356669 12210 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:56.356724 12210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:56.356925 12210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:56.358580 12210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "5fbb448689c949d98dec8e37cf5c3478"
format_stamp: "Formatted at 2024-05-02 13:59:56 on dist-test-slave-h6v0"
I20240502 13:59:56.359045 12210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "5fbb448689c949d98dec8e37cf5c3478"
format_stamp: "Formatted at 2024-05-02 13:59:56 on dist-test-slave-h6v0"
I20240502 13:59:56.362154 12210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 13:59:56.364300 12224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:56.365310 12210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:56.365420 12210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "5fbb448689c949d98dec8e37cf5c3478"
format_stamp: "Formatted at 2024-05-02 13:59:56 on dist-test-slave-h6v0"
I20240502 13:59:56.365505 12210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:56.381135 12210 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:56.381707 12210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:56.381850 12210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:56.382335 12210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:56.383167 12210 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:56.383214 12210 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:56.383277 12210 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:56.383303 12210 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:56.391254 12210 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.133:44919
I20240502 13:59:56.391289 12337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.133:44919 every 8 connection(s)
I20240502 13:59:56.392181 12210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 13:59:56.394157 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 12210
I20240502 13:59:56.394263 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 13:59:56.396270 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.134:0
--local_ip_for_outbound_sockets=127.8.86.134
--webserver_interface=127.8.86.134
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44201
--builtin_ntp_servers=127.8.86.148:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 13:59:56.400900 12338 heartbeater.cc:347] Connected to a master server at 127.8.86.190:44201
I20240502 13:59:56.401118 12338 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:56.401614 12338 heartbeater.cc:511] Master 127.8.86.190:44201 requested a full tablet report, sending...
I20240502 13:59:56.402570 11626 ts_manager.cc:194] Registered new tserver with Master: 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133:44919)
I20240502 13:59:56.403081 11626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.133:44373
W20240502 13:59:56.504374 12341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:56.504554 12341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:56.504618 12341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:56.508530 12341 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:56.508620 12341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:56.508718 12341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.134
I20240502 13:59:56.512892 12341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.8.86.134
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44201
--never_fsync=true
--heap_profile_path=/tmp/kudu.12341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.134
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:56.513880 12341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:56.515233 12341 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:56.523725 12347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:56.524451 12346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:56.525151 12349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:56.525964 12341 server_base.cc:1034] running on GCE node
I20240502 13:59:56.526279 12341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:56.526859 12341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:56.527998 12341 hybrid_clock.cc:648] HybridClock initialized: now 1714658396527986 us; error 26 us; skew 500 ppm
I20240502 13:59:56.529642 12341 webserver.cc:460] Webserver started at http://127.8.86.134:33731/ using document root <none> and password file <none>
I20240502 13:59:56.530176 12341 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:56.530227 12341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:56.530424 12341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:56.531864 12341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "a36d5a88b1a34006bd3d040f5a715766"
format_stamp: "Formatted at 2024-05-02 13:59:56 on dist-test-slave-h6v0"
I20240502 13:59:56.532272 12341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "a36d5a88b1a34006bd3d040f5a715766"
format_stamp: "Formatted at 2024-05-02 13:59:56 on dist-test-slave-h6v0"
I20240502 13:59:56.535167 12341 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240502 13:59:56.537030 12355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:56.537952 12341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:56.538050 12341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "a36d5a88b1a34006bd3d040f5a715766"
format_stamp: "Formatted at 2024-05-02 13:59:56 on dist-test-slave-h6v0"
I20240502 13:59:56.538125 12341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:56.569859 12341 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:56.570552 12341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:56.570696 12341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:56.571192 12341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:56.571990 12341 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:56.572034 12341 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:56.572085 12341 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:56.572114 12341 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:56.579519 12341 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.134:36541
I20240502 13:59:56.579573 12468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.134:36541 every 8 connection(s)
I20240502 13:59:56.580435 12341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 13:59:56.582311 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 12341
I20240502 13:59:56.582405 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 13:59:56.584403 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.135:0
--local_ip_for_outbound_sockets=127.8.86.135
--webserver_interface=127.8.86.135
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44201
--builtin_ntp_servers=127.8.86.148:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 13:59:56.588806 12469 heartbeater.cc:347] Connected to a master server at 127.8.86.190:44201
I20240502 13:59:56.589066 12469 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:56.589700 12469 heartbeater.cc:511] Master 127.8.86.190:44201 requested a full tablet report, sending...
I20240502 13:59:56.590718 11626 ts_manager.cc:194] Registered new tserver with Master: a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134:36541)
I20240502 13:59:56.591259 11626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.134:37681
W20240502 13:59:56.685909 12472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:56.686110 12472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:56.686151 12472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:56.689039 12472 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:56.689095 12472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:56.689172 12472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.135
I20240502 13:59:56.692586 12472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.8.86.135
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44201
--never_fsync=true
--heap_profile_path=/tmp/kudu.12472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.135
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:56.693375 12472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:56.694429 12472 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:56.702741 12480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:56.703222 12478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:56.704044 12477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:56.704766 12472 server_base.cc:1034] running on GCE node
I20240502 13:59:56.705157 12472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:56.705736 12472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:56.706876 12472 hybrid_clock.cc:648] HybridClock initialized: now 1714658396706854 us; error 34 us; skew 500 ppm
I20240502 13:59:56.708814 12472 webserver.cc:460] Webserver started at http://127.8.86.135:38241/ using document root <none> and password file <none>
I20240502 13:59:56.709389 12472 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:56.709444 12472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:56.709597 12472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:56.711108 12472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "7553d1bab6f642ac8395c928da1c58d7"
format_stamp: "Formatted at 2024-05-02 13:59:56 on dist-test-slave-h6v0"
I20240502 13:59:56.711534 12472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "7553d1bab6f642ac8395c928da1c58d7"
format_stamp: "Formatted at 2024-05-02 13:59:56 on dist-test-slave-h6v0"
I20240502 13:59:56.714581 12472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 13:59:56.716498 12486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:56.716854 11814 heartbeater.cc:503] Master 127.8.86.190:44201 was elected leader, sending a full tablet report...
I20240502 13:59:56.717594 12472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:56.717721 12472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "7553d1bab6f642ac8395c928da1c58d7"
format_stamp: "Formatted at 2024-05-02 13:59:56 on dist-test-slave-h6v0"
I20240502 13:59:56.717801 12472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:56.738149 12472 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:56.738756 12472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:56.738906 12472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:56.739398 12472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:56.740247 12472 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:56.740294 12472 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:56.740356 12472 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:56.740386 12472 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:56.748605 12472 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.135:34019
I20240502 13:59:56.748652 12599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.135:34019 every 8 connection(s)
I20240502 13:59:56.749475 12472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 13:59:56.749873 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 12472
I20240502 13:59:56.749989 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 13:59:56.757725 12600 heartbeater.cc:347] Connected to a master server at 127.8.86.190:44201
I20240502 13:59:56.757936 12600 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:56.758453 12600 heartbeater.cc:511] Master 127.8.86.190:44201 requested a full tablet report, sending...
I20240502 13:59:56.759335 11626 ts_manager.cc:194] Registered new tserver with Master: 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135:34019)
I20240502 13:59:56.759855 11626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.135:49971
I20240502 13:59:56.763926 8538 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 13:59:56.771960 11626 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47428:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 13:59:56.787376 11748 tablet_service.cc:1467] Processing CreateTablet for tablet 03409b5f413f499c919f2c6a5a4126f7 (DEFAULT_TABLE table=TestTable [id=977ef0b76afd4ccc955a0f354e4f5725]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:56.787376 11879 tablet_service.cc:1467] Processing CreateTablet for tablet 03409b5f413f499c919f2c6a5a4126f7 (DEFAULT_TABLE table=TestTable [id=977ef0b76afd4ccc955a0f354e4f5725]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:56.788343 11879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03409b5f413f499c919f2c6a5a4126f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:56.788342 11748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03409b5f413f499c919f2c6a5a4126f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:56.794051 12616 tablet_bootstrap.cc:492] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9: Bootstrap starting.
I20240502 13:59:56.794052 12617 tablet_bootstrap.cc:492] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea: Bootstrap starting.
I20240502 13:59:56.796209 12617 tablet_bootstrap.cc:654] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:56.796209 12616 tablet_bootstrap.cc:654] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:56.797082 12617 log.cc:824] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:56.797082 12616 log.cc:824] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:56.798564 12617 tablet_bootstrap.cc:492] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea: No bootstrap required, opened a new log
I20240502 13:59:56.798586 12616 tablet_bootstrap.cc:492] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9: No bootstrap required, opened a new log
I20240502 13:59:56.798730 12616 ts_tablet_manager.cc:1367] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 13:59:56.798722 12617 ts_tablet_manager.cc:1367] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 13:59:56.801008 12616 raft_consensus.cc:348] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 13:59:56.801177 12616 raft_consensus.cc:374] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:56.801213 12616 raft_consensus.cc:729] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Initialized, Role: FOLLOWER
I20240502 13:59:56.801294 12617 raft_consensus.cc:348] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 13:59:56.801450 12617 raft_consensus.cc:374] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:56.801509 12617 raft_consensus.cc:729] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Initialized, Role: FOLLOWER
I20240502 13:59:56.801625 12616 consensus_queue.cc:260] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 13:59:56.801851 12617 consensus_queue.cc:260] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 13:59:56.802255 11945 heartbeater.cc:503] Master 127.8.86.190:44201 was elected leader, sending a full tablet report...
I20240502 13:59:56.802697 12617 ts_tablet_manager.cc:1398] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 13:59:56.803449 12616 ts_tablet_manager.cc:1398] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20240502 13:59:56.877317 11946 tablet.cc:2369] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 13:59:56.954641 11815 tablet.cc:2369] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 13:59:57.060472 12076 heartbeater.cc:503] Master 127.8.86.190:44201 was elected leader, sending a full tablet report...
I20240502 13:59:57.200008 12620 raft_consensus.cc:482] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:57.200241 12620 raft_consensus.cc:504] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 13:59:57.201217 12620 leader_election.cc:288] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 13:59:57.204684 11768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03409b5f413f499c919f2c6a5a4126f7" candidate_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" is_pre_election: true
I20240502 13:59:57.204955 11768 raft_consensus.cc:2454] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbf92f56d7144a22a6d482c73c7c7ea9 in term 0.
I20240502 13:59:57.205361 11834 leader_election.cc:302] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 73ce3d001cdf4d0c8e1487958b7694ea, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 13:59:57.205642 12620 raft_consensus.cc:2781] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:57.205715 12620 raft_consensus.cc:482] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:57.205780 12620 raft_consensus.cc:3037] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:57.206686 12620 raft_consensus.cc:504] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 13:59:57.206941 12620 leader_election.cc:288] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 election: Requested vote from peers 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 13:59:57.207298 11768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03409b5f413f499c919f2c6a5a4126f7" candidate_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
I20240502 13:59:57.207401 11768 raft_consensus.cc:3037] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:57.208220 11768 raft_consensus.cc:2454] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbf92f56d7144a22a6d482c73c7c7ea9 in term 1.
I20240502 13:59:57.208519 11834 leader_election.cc:302] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 73ce3d001cdf4d0c8e1487958b7694ea, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 13:59:57.208673 12620 raft_consensus.cc:2781] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:57.208874 12620 raft_consensus.cc:686] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Becoming Leader. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Running, Role: LEADER
I20240502 13:59:57.209136 12620 consensus_queue.cc:237] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 13:59:57.211967 11626 catalog_manager.cc:5484] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:57.229115 8538 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 13:59:57.230374 8538 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 73ce3d001cdf4d0c8e1487958b7694ea to finish bootstrapping
I20240502 13:59:57.231379 12207 heartbeater.cc:503] Master 127.8.86.190:44201 was elected leader, sending a full tablet report...
I20240502 13:59:57.235689 8538 ts_itest-base.cc:233] Waiting for 1 tablets on tserver cbf92f56d7144a22a6d482c73c7c7ea9 to finish bootstrapping
I20240502 13:59:57.239876 8538 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 65a078666d30488f9c204d93e6394ef9 to finish bootstrapping
I20240502 13:59:57.245151 8538 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4f6f2d2b1569415cbbbea2466dc28e28 to finish bootstrapping
I20240502 13:59:57.250368 8538 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5fbb448689c949d98dec8e37cf5c3478 to finish bootstrapping
I20240502 13:59:57.255378 8538 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a36d5a88b1a34006bd3d040f5a715766 to finish bootstrapping
I20240502 13:59:57.260458 8538 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7553d1bab6f642ac8395c928da1c58d7 to finish bootstrapping
I20240502 13:59:57.265286 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 12079
I20240502 13:59:57.271502 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 12210
I20240502 13:59:57.277490 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 12341
I20240502 13:59:57.282985 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 12472
I20240502 13:59:57.776621 12631 consensus_queue.cc:1035] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:01.039405 11626 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47428:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:01.046164 11748 tablet_service.cc:1467] Processing CreateTablet for tablet c951304cdaa54040a81e51d076fdeaaf (DEFAULT_TABLE table=rebalance_test_table_0 [id=610fb42d93954513952217d0731890d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:01.046525 11748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c951304cdaa54040a81e51d076fdeaaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:01.046591 11747 tablet_service.cc:1467] Processing CreateTablet for tablet 49041719e0ae42a390452b8716cc3ca7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=610fb42d93954513952217d0731890d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:01.046900 11747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49041719e0ae42a390452b8716cc3ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:01.049106 12645 tablet_bootstrap.cc:492] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea: Bootstrap starting.
I20240502 14:00:01.049196 11879 tablet_service.cc:1467] Processing CreateTablet for tablet db4ce5e277504649a781e23861baa667 (DEFAULT_TABLE table=rebalance_test_table_0 [id=610fb42d93954513952217d0731890d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:01.049522 11879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4ce5e277504649a781e23861baa667. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:01.050125 12645 tablet_bootstrap.cc:654] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:01.051790 12645 tablet_bootstrap.cc:492] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea: No bootstrap required, opened a new log
I20240502 14:00:01.051879 12645 ts_tablet_manager.cc:1367] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:00:01.051995 12647 tablet_bootstrap.cc:492] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9: Bootstrap starting.
I20240502 14:00:01.052426 12645 raft_consensus.cc:348] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.052567 12645 raft_consensus.cc:374] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:01.052596 12645 raft_consensus.cc:729] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Initialized, Role: FOLLOWER
I20240502 14:00:01.052727 12645 consensus_queue.cc:260] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.052841 12647 tablet_bootstrap.cc:654] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:01.053027 12645 ts_tablet_manager.cc:1398] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:01.053185 12645 tablet_bootstrap.cc:492] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea: Bootstrap starting.
I20240502 14:00:01.053423 12009 tablet_service.cc:1467] Processing CreateTablet for tablet db4ce5e277504649a781e23861baa667 (DEFAULT_TABLE table=rebalance_test_table_0 [id=610fb42d93954513952217d0731890d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:01.053426 12010 tablet_service.cc:1467] Processing CreateTablet for tablet c951304cdaa54040a81e51d076fdeaaf (DEFAULT_TABLE table=rebalance_test_table_0 [id=610fb42d93954513952217d0731890d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:01.054011 12645 tablet_bootstrap.cc:654] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:01.054199 12008 tablet_service.cc:1467] Processing CreateTablet for tablet 49041719e0ae42a390452b8716cc3ca7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=610fb42d93954513952217d0731890d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:01.054419 12009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4ce5e277504649a781e23861baa667. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:01.054792 12008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49041719e0ae42a390452b8716cc3ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:01.055899 12010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c951304cdaa54040a81e51d076fdeaaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:01.055977 12645 tablet_bootstrap.cc:492] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea: No bootstrap required, opened a new log
I20240502 14:00:01.056063 12645 ts_tablet_manager.cc:1367] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:00:01.056576 12645 raft_consensus.cc:348] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.056748 12645 raft_consensus.cc:374] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:01.056795 12645 raft_consensus.cc:729] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Initialized, Role: FOLLOWER
I20240502 14:00:01.056910 12645 consensus_queue.cc:260] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.057191 12645 ts_tablet_manager.cc:1398] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:01.059849 12647 tablet_bootstrap.cc:492] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9: No bootstrap required, opened a new log
I20240502 14:00:01.059962 12647 ts_tablet_manager.cc:1367] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:01.060499 12647 raft_consensus.cc:348] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.060659 12647 raft_consensus.cc:374] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:01.060743 12647 raft_consensus.cc:729] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Initialized, Role: FOLLOWER
I20240502 14:00:01.060901 12647 consensus_queue.cc:260] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.061259 12651 tablet_bootstrap.cc:492] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9: Bootstrap starting.
I20240502 14:00:01.061302 12647 ts_tablet_manager.cc:1398] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:01.063607 12651 tablet_bootstrap.cc:654] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:01.064607 12651 log.cc:824] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:01.066187 12651 tablet_bootstrap.cc:492] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9: No bootstrap required, opened a new log
I20240502 14:00:01.066381 12651 ts_tablet_manager.cc:1367] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:00:01.069063 12651 raft_consensus.cc:348] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.069304 12651 raft_consensus.cc:374] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:01.069402 12651 raft_consensus.cc:729] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a078666d30488f9c204d93e6394ef9, State: Initialized, Role: FOLLOWER
I20240502 14:00:01.069854 12651 consensus_queue.cc:260] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.070572 12651 ts_tablet_manager.cc:1398] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:00:01.070794 12651 tablet_bootstrap.cc:492] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9: Bootstrap starting.
I20240502 14:00:01.071564 12651 tablet_bootstrap.cc:654] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:01.072571 12651 tablet_bootstrap.cc:492] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9: No bootstrap required, opened a new log
I20240502 14:00:01.072654 12651 ts_tablet_manager.cc:1367] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:01.073071 12651 raft_consensus.cc:348] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.073153 12651 raft_consensus.cc:374] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:01.073186 12651 raft_consensus.cc:729] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a078666d30488f9c204d93e6394ef9, State: Initialized, Role: FOLLOWER
I20240502 14:00:01.073308 12651 consensus_queue.cc:260] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.073508 12651 ts_tablet_manager.cc:1398] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:01.073673 12651 tablet_bootstrap.cc:492] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9: Bootstrap starting.
I20240502 14:00:01.074530 12651 tablet_bootstrap.cc:654] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:01.075675 12651 tablet_bootstrap.cc:492] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9: No bootstrap required, opened a new log
I20240502 14:00:01.075773 12651 ts_tablet_manager.cc:1367] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:01.076134 12651 raft_consensus.cc:348] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.076210 12651 raft_consensus.cc:374] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:01.076231 12651 raft_consensus.cc:729] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a078666d30488f9c204d93e6394ef9, State: Initialized, Role: FOLLOWER
I20240502 14:00:01.076328 12651 consensus_queue.cc:260] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.076570 12651 ts_tablet_manager.cc:1398] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:01.194098 12654 raft_consensus.cc:482] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:01.194356 12654 raft_consensus.cc:504] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.195402 12654 leader_election.cc:288] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 14:00:01.199584 11768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49041719e0ae42a390452b8716cc3ca7" candidate_uuid: "65a078666d30488f9c204d93e6394ef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" is_pre_election: true
I20240502 14:00:01.199738 11768 raft_consensus.cc:2454] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65a078666d30488f9c204d93e6394ef9 in term 0.
I20240502 14:00:01.200126 11965 leader_election.cc:302] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, 73ce3d001cdf4d0c8e1487958b7694ea; no voters:
I20240502 14:00:01.200376 12654 raft_consensus.cc:2781] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:01.200445 12654 raft_consensus.cc:482] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:01.200476 12654 raft_consensus.cc:3037] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:01.201346 12654 raft_consensus.cc:504] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.201628 12654 leader_election.cc:288] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [CANDIDATE]: Term 1 election: Requested vote from peers 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 14:00:01.201993 11768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49041719e0ae42a390452b8716cc3ca7" candidate_uuid: "65a078666d30488f9c204d93e6394ef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
I20240502 14:00:01.202097 11768 raft_consensus.cc:3037] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:01.203027 11768 raft_consensus.cc:2454] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65a078666d30488f9c204d93e6394ef9 in term 1.
I20240502 14:00:01.203315 11965 leader_election.cc:302] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, 73ce3d001cdf4d0c8e1487958b7694ea; no voters:
I20240502 14:00:01.203490 12654 raft_consensus.cc:2781] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:01.203670 12654 raft_consensus.cc:686] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Becoming Leader. State: Replica: 65a078666d30488f9c204d93e6394ef9, State: Running, Role: LEADER
I20240502 14:00:01.203948 12654 consensus_queue.cc:237] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.206597 11626 catalog_manager.cc:5484] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65a078666d30488f9c204d93e6394ef9 (127.8.86.131). New cstate: current_term: 1 leader_uuid: "65a078666d30488f9c204d93e6394ef9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: HEALTHY } } }
I20240502 14:00:01.223847 12654 raft_consensus.cc:482] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:01.223980 12654 raft_consensus.cc:504] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.224413 12654 leader_election.cc:288] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377)
I20240502 14:00:01.227707 11899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ce5e277504649a781e23861baa667" candidate_uuid: "65a078666d30488f9c204d93e6394ef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" is_pre_election: true
I20240502 14:00:01.227943 11899 raft_consensus.cc:2454] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65a078666d30488f9c204d93e6394ef9 in term 0.
I20240502 14:00:01.228359 11966 leader_election.cc:302] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:01.228533 12654 raft_consensus.cc:2781] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:01.228595 12654 raft_consensus.cc:482] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:01.228636 12654 raft_consensus.cc:3037] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:01.229393 12654 raft_consensus.cc:504] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.229662 12654 leader_election.cc:288] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [CANDIDATE]: Term 1 election: Requested vote from peers cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377)
I20240502 14:00:01.230041 11899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ce5e277504649a781e23861baa667" candidate_uuid: "65a078666d30488f9c204d93e6394ef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9"
I20240502 14:00:01.230149 11899 raft_consensus.cc:3037] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:01.230924 11899 raft_consensus.cc:2454] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65a078666d30488f9c204d93e6394ef9 in term 1.
I20240502 14:00:01.231199 11966 leader_election.cc:302] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:01.231350 12654 raft_consensus.cc:2781] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:01.231431 12654 raft_consensus.cc:686] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Becoming Leader. State: Replica: 65a078666d30488f9c204d93e6394ef9, State: Running, Role: LEADER
I20240502 14:00:01.231555 12654 consensus_queue.cc:237] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.233106 11626 catalog_manager.cc:5484] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65a078666d30488f9c204d93e6394ef9 (127.8.86.131). New cstate: current_term: 1 leader_uuid: "65a078666d30488f9c204d93e6394ef9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: HEALTHY } } }
W20240502 14:00:01.301170 12077 tablet.cc:2369] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:01.392191 12648 raft_consensus.cc:482] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:01.392359 12648 raft_consensus.cc:504] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.393309 12648 leader_election.cc:288] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:01.396868 12030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c951304cdaa54040a81e51d076fdeaaf" candidate_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a078666d30488f9c204d93e6394ef9" is_pre_election: true
I20240502 14:00:01.397055 12030 raft_consensus.cc:2454] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73ce3d001cdf4d0c8e1487958b7694ea in term 0.
I20240502 14:00:01.397496 11703 leader_election.cc:302] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, 73ce3d001cdf4d0c8e1487958b7694ea; no voters:
I20240502 14:00:01.397747 12648 raft_consensus.cc:2781] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:01.397805 12648 raft_consensus.cc:482] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:01.397832 12648 raft_consensus.cc:3037] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:01.398692 12648 raft_consensus.cc:504] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.398947 12648 leader_election.cc:288] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 election: Requested vote from peers 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:01.399379 12030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c951304cdaa54040a81e51d076fdeaaf" candidate_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a078666d30488f9c204d93e6394ef9"
I20240502 14:00:01.399468 12030 raft_consensus.cc:3037] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:01.400158 12030 raft_consensus.cc:2454] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73ce3d001cdf4d0c8e1487958b7694ea in term 1.
I20240502 14:00:01.400420 11703 leader_election.cc:302] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, 73ce3d001cdf4d0c8e1487958b7694ea; no voters:
I20240502 14:00:01.400585 12648 raft_consensus.cc:2781] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:01.400794 12648 raft_consensus.cc:686] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Becoming Leader. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Running, Role: LEADER
I20240502 14:00:01.401085 12648 consensus_queue.cc:237] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.403005 11626 catalog_manager.cc:5484] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129). New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: UNKNOWN } } }
W20240502 14:00:01.514655 12666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:01.514887 12666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:01.517840 12666 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 14:00:01.608191 11768 raft_consensus.cc:1261] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:01.608407 11899 raft_consensus.cc:1261] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:01.608990 12659 consensus_queue.cc:1035] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:01.609385 12659 consensus_queue.cc:1035] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:01.613886 12694 mvcc.cc:205] Tried to move back new op lower bound from 7023240812981227520 to 7023240811333337088. Current Snapshot: MvccSnapshot[applied={T|T < 7023240812981227520}]
I20240502 14:00:01.615223 12030 raft_consensus.cc:1261] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:01.616102 12648 consensus_queue.cc:1035] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:01.616652 12697 mvcc.cc:205] Tried to move back new op lower bound from 7023240812981227520 to 7023240811333337088. Current Snapshot: MvccSnapshot[applied={T|T < 7023240812981227520}]
I20240502 14:00:01.626475 12699 mvcc.cc:205] Tried to move back new op lower bound from 7023240812981927936 to 7023240811445469184. Current Snapshot: MvccSnapshot[applied={T|T < 7023240812981927936}]
INSERT report
rows total: 2000
time total: 91.6735 ms
time per row: 0.0458367 ms
I20240502 14:00:01.654832 11620 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47428:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:01.660089 11748 tablet_service.cc:1467] Processing CreateTablet for tablet 3a50cea2712d48ca812cea7983d16840 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6a2f7f1e4dc45ffbe7f9f2fba15577e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:01.660255 11878 tablet_service.cc:1467] Processing CreateTablet for tablet 986b926ead014774925e3b8a3892c56d (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6a2f7f1e4dc45ffbe7f9f2fba15577e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:01.660255 11879 tablet_service.cc:1467] Processing CreateTablet for tablet 3a50cea2712d48ca812cea7983d16840 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6a2f7f1e4dc45ffbe7f9f2fba15577e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:01.660431 11748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a50cea2712d48ca812cea7983d16840. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:01.660557 11878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986b926ead014774925e3b8a3892c56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:01.660676 11879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a50cea2712d48ca812cea7983d16840. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:01.662398 12713 tablet_bootstrap.cc:492] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea: Bootstrap starting.
I20240502 14:00:01.662470 11747 tablet_service.cc:1467] Processing CreateTablet for tablet b9a6f7fca259430dac2ca17a8b180b2a (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6a2f7f1e4dc45ffbe7f9f2fba15577e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:01.662540 11877 tablet_service.cc:1467] Processing CreateTablet for tablet b9a6f7fca259430dac2ca17a8b180b2a (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6a2f7f1e4dc45ffbe7f9f2fba15577e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:01.662667 11747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9a6f7fca259430dac2ca17a8b180b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:01.662765 11877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9a6f7fca259430dac2ca17a8b180b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:01.663264 12713 tablet_bootstrap.cc:654] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:01.663530 12714 tablet_bootstrap.cc:492] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9: Bootstrap starting.
I20240502 14:00:01.664405 12714 tablet_bootstrap.cc:654] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:01.664830 12008 tablet_service.cc:1467] Processing CreateTablet for tablet 986b926ead014774925e3b8a3892c56d (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6a2f7f1e4dc45ffbe7f9f2fba15577e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:01.665119 12008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986b926ead014774925e3b8a3892c56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:01.665838 12713 tablet_bootstrap.cc:492] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea: No bootstrap required, opened a new log
I20240502 14:00:01.665896 12713 ts_tablet_manager.cc:1367] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:00:01.666234 12713 raft_consensus.cc:348] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:01.666373 12713 raft_consensus.cc:374] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:01.666440 12713 raft_consensus.cc:729] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Initialized, Role: FOLLOWER
I20240502 14:00:01.666603 12713 consensus_queue.cc:260] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:01.666970 12713 ts_tablet_manager.cc:1398] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:01.666993 12714 tablet_bootstrap.cc:492] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9: No bootstrap required, opened a new log
I20240502 14:00:01.667173 12713 tablet_bootstrap.cc:492] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea: Bootstrap starting.
I20240502 14:00:01.667192 12714 ts_tablet_manager.cc:1367] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:01.668012 12713 tablet_bootstrap.cc:654] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:01.668352 12714 raft_consensus.cc:348] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:01.668442 12714 raft_consensus.cc:374] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:01.668476 12714 raft_consensus.cc:729] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Initialized, Role: FOLLOWER
I20240502 14:00:01.668653 12714 consensus_queue.cc:260] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:01.669016 12713 tablet_bootstrap.cc:492] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea: No bootstrap required, opened a new log
I20240502 14:00:01.669081 12713 ts_tablet_manager.cc:1367] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:00:01.669399 12713 raft_consensus.cc:348] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:01.669489 12713 raft_consensus.cc:374] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:01.669515 12713 raft_consensus.cc:729] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Initialized, Role: FOLLOWER
I20240502 14:00:01.669627 12713 consensus_queue.cc:260] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:01.669850 12714 ts_tablet_manager.cc:1398] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:01.669857 12713 ts_tablet_manager.cc:1398] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:01.670025 12714 tablet_bootstrap.cc:492] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9: Bootstrap starting.
I20240502 14:00:01.671150 12714 tablet_bootstrap.cc:654] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:01.671890 12717 tablet_bootstrap.cc:492] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9: Bootstrap starting.
I20240502 14:00:01.672397 12714 tablet_bootstrap.cc:492] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9: No bootstrap required, opened a new log
I20240502 14:00:01.672475 12714 ts_tablet_manager.cc:1367] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:01.672817 12714 raft_consensus.cc:348] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:01.672894 12714 raft_consensus.cc:374] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:01.672926 12714 raft_consensus.cc:729] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Initialized, Role: FOLLOWER
I20240502 14:00:01.673044 12714 consensus_queue.cc:260] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:01.673238 12714 ts_tablet_manager.cc:1398] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:01.673383 12714 tablet_bootstrap.cc:492] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9: Bootstrap starting.
I20240502 14:00:01.674052 12714 tablet_bootstrap.cc:654] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:01.674997 12714 tablet_bootstrap.cc:492] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9: No bootstrap required, opened a new log
I20240502 14:00:01.675078 12714 ts_tablet_manager.cc:1367] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:01.675407 12714 raft_consensus.cc:348] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.675484 12714 raft_consensus.cc:374] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:01.675518 12714 raft_consensus.cc:729] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Initialized, Role: FOLLOWER
I20240502 14:00:01.675643 12714 consensus_queue.cc:260] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.675936 12714 ts_tablet_manager.cc:1398] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:01.676545 12717 tablet_bootstrap.cc:654] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:01.677659 12717 tablet_bootstrap.cc:492] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9: No bootstrap required, opened a new log
I20240502 14:00:01.677764 12717 ts_tablet_manager.cc:1367] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:00:01.678184 12717 raft_consensus.cc:348] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.678320 12717 raft_consensus.cc:374] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:01.678364 12717 raft_consensus.cc:729] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a078666d30488f9c204d93e6394ef9, State: Initialized, Role: FOLLOWER
I20240502 14:00:01.678465 12717 consensus_queue.cc:260] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.678697 12717 ts_tablet_manager.cc:1398] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:01.687889 12696 raft_consensus.cc:482] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:01.687983 12696 raft_consensus.cc:504] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.688319 12696 leader_election.cc:288] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:01.691388 12030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986b926ead014774925e3b8a3892c56d" candidate_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a078666d30488f9c204d93e6394ef9" is_pre_election: true
I20240502 14:00:01.691504 12030 raft_consensus.cc:2454] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbf92f56d7144a22a6d482c73c7c7ea9 in term 0.
I20240502 14:00:01.691751 11834 leader_election.cc:302] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:01.691896 12696 raft_consensus.cc:2781] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:01.691953 12696 raft_consensus.cc:482] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:01.691985 12696 raft_consensus.cc:3037] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:01.692793 12696 raft_consensus.cc:504] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.693042 12696 leader_election.cc:288] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 election: Requested vote from peers 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:01.693323 12030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986b926ead014774925e3b8a3892c56d" candidate_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a078666d30488f9c204d93e6394ef9"
I20240502 14:00:01.693413 12030 raft_consensus.cc:3037] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:01.694057 12030 raft_consensus.cc:2454] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbf92f56d7144a22a6d482c73c7c7ea9 in term 1.
I20240502 14:00:01.694340 11834 leader_election.cc:302] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:01.694479 12696 raft_consensus.cc:2781] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:01.694640 12696 raft_consensus.cc:686] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Becoming Leader. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Running, Role: LEADER
I20240502 14:00:01.694770 12696 consensus_queue.cc:237] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:01.696064 11625 catalog_manager.cc:5484] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:01.887565 12664 raft_consensus.cc:482] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:01.887729 12664 raft_consensus.cc:504] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:01.888177 12664 leader_election.cc:288] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377)
I20240502 14:00:01.891614 11899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a50cea2712d48ca812cea7983d16840" candidate_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" is_pre_election: true
I20240502 14:00:01.891754 11899 raft_consensus.cc:2454] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73ce3d001cdf4d0c8e1487958b7694ea in term 0.
I20240502 14:00:01.892112 11704 leader_election.cc:302] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 73ce3d001cdf4d0c8e1487958b7694ea, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:01.892331 12664 raft_consensus.cc:2781] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:01.892388 12664 raft_consensus.cc:482] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:01.892419 12664 raft_consensus.cc:3037] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:01.974222 12664 raft_consensus.cc:504] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:01.974766 12664 leader_election.cc:288] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 election: Requested vote from peers cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377)
I20240502 14:00:01.975387 11899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a50cea2712d48ca812cea7983d16840" candidate_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9"
I20240502 14:00:01.975534 11899 raft_consensus.cc:3037] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:01.976735 11899 raft_consensus.cc:2454] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73ce3d001cdf4d0c8e1487958b7694ea in term 1.
I20240502 14:00:01.977136 11704 leader_election.cc:302] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 73ce3d001cdf4d0c8e1487958b7694ea, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:01.977325 12664 raft_consensus.cc:2781] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:01.977433 12664 raft_consensus.cc:686] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Becoming Leader. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Running, Role: LEADER
I20240502 14:00:01.977608 12664 consensus_queue.cc:237] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:01.979266 11625 catalog_manager.cc:5484] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129). New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:02.079674 12696 raft_consensus.cc:482] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:02.079851 12664 raft_consensus.cc:482] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:02.079859 12696 raft_consensus.cc:504] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:02.079981 12664 raft_consensus.cc:504] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:02.080263 12696 leader_election.cc:288] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 14:00:02.080329 12664 leader_election.cc:288] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377)
I20240502 14:00:02.080863 11768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a6f7fca259430dac2ca17a8b180b2a" candidate_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" is_pre_election: true
I20240502 14:00:02.080917 11899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a6f7fca259430dac2ca17a8b180b2a" candidate_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" is_pre_election: true
I20240502 14:00:02.081017 11899 raft_consensus.cc:2454] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73ce3d001cdf4d0c8e1487958b7694ea in term 0.
I20240502 14:00:02.081017 11768 raft_consensus.cc:2454] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbf92f56d7144a22a6d482c73c7c7ea9 in term 0.
I20240502 14:00:02.081308 11834 leader_election.cc:302] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 73ce3d001cdf4d0c8e1487958b7694ea, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:02.081311 11704 leader_election.cc:302] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 73ce3d001cdf4d0c8e1487958b7694ea, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:02.081499 12696 raft_consensus.cc:2781] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:02.081547 12664 raft_consensus.cc:2781] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:02.081566 12696 raft_consensus.cc:482] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:02.081596 12664 raft_consensus.cc:482] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:02.081619 12696 raft_consensus.cc:3037] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:02.081643 12664 raft_consensus.cc:3037] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:02.171998 12696 raft_consensus.cc:504] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:02.171986 12664 raft_consensus.cc:504] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:02.172369 12696 leader_election.cc:288] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 election: Requested vote from peers 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 14:00:02.172669 12664 leader_election.cc:288] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 election: Requested vote from peers cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377)
I20240502 14:00:02.172741 11899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a6f7fca259430dac2ca17a8b180b2a" candidate_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9"
I20240502 14:00:02.172746 11768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a6f7fca259430dac2ca17a8b180b2a" candidate_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
I20240502 14:00:02.172917 11899 raft_consensus.cc:2379] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 73ce3d001cdf4d0c8e1487958b7694ea in current term 1: Already voted for candidate cbf92f56d7144a22a6d482c73c7c7ea9 in this term.
I20240502 14:00:02.172942 11768 raft_consensus.cc:2379] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cbf92f56d7144a22a6d482c73c7c7ea9 in current term 1: Already voted for candidate 73ce3d001cdf4d0c8e1487958b7694ea in this term.
I20240502 14:00:02.173251 11704 leader_election.cc:302] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 73ce3d001cdf4d0c8e1487958b7694ea; no voters: cbf92f56d7144a22a6d482c73c7c7ea9
I20240502 14:00:02.173225 11834 leader_election.cc:302] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: cbf92f56d7144a22a6d482c73c7c7ea9; no voters: 73ce3d001cdf4d0c8e1487958b7694ea
I20240502 14:00:02.173447 12696 raft_consensus.cc:2726] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:00:02.173480 12664 raft_consensus.cc:2726] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:00:02.209698 12696 consensus_queue.cc:1035] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:02.452257 12664 consensus_queue.cc:1035] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:03.782231 12664 raft_consensus.cc:482] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:03.782395 12664 raft_consensus.cc:504] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:03.782792 12664 leader_election.cc:288] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377)
I20240502 14:00:03.783290 11899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a6f7fca259430dac2ca17a8b180b2a" candidate_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" is_pre_election: true
I20240502 14:00:03.783419 11899 raft_consensus.cc:2454] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73ce3d001cdf4d0c8e1487958b7694ea in term 1.
I20240502 14:00:03.783749 11704 leader_election.cc:302] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 73ce3d001cdf4d0c8e1487958b7694ea, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:03.783926 12664 raft_consensus.cc:2781] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:00:03.783987 12664 raft_consensus.cc:482] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:03.784013 12664 raft_consensus.cc:3037] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:03.784837 12664 raft_consensus.cc:504] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:03.785100 12664 leader_election.cc:288] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 2 election: Requested vote from peers cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377)
I20240502 14:00:03.785491 11899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a6f7fca259430dac2ca17a8b180b2a" candidate_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9"
I20240502 14:00:03.785622 11899 raft_consensus.cc:3037] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:03.786502 11899 raft_consensus.cc:2454] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73ce3d001cdf4d0c8e1487958b7694ea in term 2.
I20240502 14:00:03.786783 11704 leader_election.cc:302] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 73ce3d001cdf4d0c8e1487958b7694ea, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:03.786935 12664 raft_consensus.cc:2781] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:03.787109 12664 raft_consensus.cc:686] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 LEADER]: Becoming Leader. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Running, Role: LEADER
I20240502 14:00:03.787258 12664 consensus_queue.cc:237] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:03.788719 11620 catalog_manager.cc:5484] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: term changed from 0 to 2, leader changed from <none> to 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129). New cstate: current_term: 2 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } }
I20240502 14:00:04.182417 11898 raft_consensus.cc:1261] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 14:00:04.182935 12664 consensus_queue.cc:1035] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:00:04.376282 12745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:04.376540 12745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:04.379539 12745 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 83.4298 ms
time per row: 0.0417149 ms
I20240502 14:00:04.504602 11624 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47428:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:04.516296 11747 tablet_service.cc:1467] Processing CreateTablet for tablet c0230720456b49339caaa9d313b6ce64 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d77f8a4d2b7c4f42901744c7061e0d56]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:04.516479 12008 tablet_service.cc:1467] Processing CreateTablet for tablet c0230720456b49339caaa9d313b6ce64 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d77f8a4d2b7c4f42901744c7061e0d56]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:04.516578 11878 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd1b1872364411e86b810420c54034a (DEFAULT_TABLE table=rebalance_test_table_2 [id=d77f8a4d2b7c4f42901744c7061e0d56]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:04.516685 11747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0230720456b49339caaa9d313b6ce64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:04.516728 12008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0230720456b49339caaa9d313b6ce64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:04.516906 11878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cd1b1872364411e86b810420c54034a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:04.518606 11748 tablet_service.cc:1467] Processing CreateTablet for tablet 11f585c28308464bb46aabdf0ad98a08 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d77f8a4d2b7c4f42901744c7061e0d56]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:04.518855 11748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f585c28308464bb46aabdf0ad98a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:04.519050 12797 tablet_bootstrap.cc:492] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9: Bootstrap starting.
I20240502 14:00:04.519753 12009 tablet_service.cc:1467] Processing CreateTablet for tablet 11f585c28308464bb46aabdf0ad98a08 (DEFAULT_TABLE table=rebalance_test_table_2 [id=d77f8a4d2b7c4f42901744c7061e0d56]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:04.519990 12797 tablet_bootstrap.cc:654] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:04.519999 12009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f585c28308464bb46aabdf0ad98a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:04.520855 12008 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd1b1872364411e86b810420c54034a (DEFAULT_TABLE table=rebalance_test_table_2 [id=d77f8a4d2b7c4f42901744c7061e0d56]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:04.520995 12799 tablet_bootstrap.cc:492] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea: Bootstrap starting.
I20240502 14:00:04.521072 12008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cd1b1872364411e86b810420c54034a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:04.521852 12799 tablet_bootstrap.cc:654] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:04.522395 12798 tablet_bootstrap.cc:492] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9: Bootstrap starting.
I20240502 14:00:04.522922 12797 tablet_bootstrap.cc:492] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9: No bootstrap required, opened a new log
I20240502 14:00:04.523068 12797 ts_tablet_manager.cc:1367] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:04.523360 12798 tablet_bootstrap.cc:654] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:04.523561 12799 tablet_bootstrap.cc:492] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea: No bootstrap required, opened a new log
I20240502 14:00:04.523728 12799 ts_tablet_manager.cc:1367] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:00:04.524297 12799 raft_consensus.cc:348] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:04.524482 12799 raft_consensus.cc:374] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:04.524564 12799 raft_consensus.cc:729] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Initialized, Role: FOLLOWER
I20240502 14:00:04.524636 12798 tablet_bootstrap.cc:492] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9: No bootstrap required, opened a new log
I20240502 14:00:04.524616 12797 raft_consensus.cc:348] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:04.524715 12798 ts_tablet_manager.cc:1367] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:04.524725 12797 raft_consensus.cc:374] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:04.524760 12797 raft_consensus.cc:729] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a078666d30488f9c204d93e6394ef9, State: Initialized, Role: FOLLOWER
I20240502 14:00:04.524858 12799 consensus_queue.cc:260] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:04.524895 12797 consensus_queue.cc:260] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:04.525285 12799 ts_tablet_manager.cc:1398] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:04.525295 12797 ts_tablet_manager.cc:1398] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:04.525470 12797 tablet_bootstrap.cc:492] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9: Bootstrap starting.
I20240502 14:00:04.525501 12799 tablet_bootstrap.cc:492] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea: Bootstrap starting.
I20240502 14:00:04.526436 12797 tablet_bootstrap.cc:654] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:04.526443 12799 tablet_bootstrap.cc:654] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:04.527540 12797 tablet_bootstrap.cc:492] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9: No bootstrap required, opened a new log
I20240502 14:00:04.527674 12797 ts_tablet_manager.cc:1367] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:04.528028 12799 tablet_bootstrap.cc:492] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea: No bootstrap required, opened a new log
I20240502 14:00:04.528101 12797 raft_consensus.cc:348] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:04.528208 12797 raft_consensus.cc:374] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:04.528254 12797 raft_consensus.cc:729] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a078666d30488f9c204d93e6394ef9, State: Initialized, Role: FOLLOWER
I20240502 14:00:04.528187 12799 ts_tablet_manager.cc:1367] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:04.528383 12797 consensus_queue.cc:260] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:04.528744 12797 ts_tablet_manager.cc:1398] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:04.528924 12797 tablet_bootstrap.cc:492] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9: Bootstrap starting.
I20240502 14:00:04.529623 12799 raft_consensus.cc:348] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:04.529711 12799 raft_consensus.cc:374] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:04.529733 12797 tablet_bootstrap.cc:654] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:04.529744 12799 raft_consensus.cc:729] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Initialized, Role: FOLLOWER
I20240502 14:00:04.529861 12799 consensus_queue.cc:260] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:04.530095 12799 ts_tablet_manager.cc:1398] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:00:04.530848 12797 tablet_bootstrap.cc:492] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9: No bootstrap required, opened a new log
I20240502 14:00:04.530941 12797 ts_tablet_manager.cc:1367] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:04.531373 12797 raft_consensus.cc:348] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:04.531456 12797 raft_consensus.cc:374] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:04.531477 12797 raft_consensus.cc:729] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a078666d30488f9c204d93e6394ef9, State: Initialized, Role: FOLLOWER
I20240502 14:00:04.531569 12797 consensus_queue.cc:260] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:04.530858 12798 raft_consensus.cc:348] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:04.531667 12798 raft_consensus.cc:374] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:04.531750 12798 raft_consensus.cc:729] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Initialized, Role: FOLLOWER
I20240502 14:00:04.531877 12797 ts_tablet_manager.cc:1398] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:04.531898 12798 consensus_queue.cc:260] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:04.532169 12798 ts_tablet_manager.cc:1398] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 14:00:04.602474 12734 raft_consensus.cc:482] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:04.602605 12734 raft_consensus.cc:504] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:04.602983 12734 leader_election.cc:288] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:04.603449 12030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f585c28308464bb46aabdf0ad98a08" candidate_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a078666d30488f9c204d93e6394ef9" is_pre_election: true
I20240502 14:00:04.603579 12030 raft_consensus.cc:2454] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73ce3d001cdf4d0c8e1487958b7694ea in term 0.
I20240502 14:00:04.603888 11703 leader_election.cc:302] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, 73ce3d001cdf4d0c8e1487958b7694ea; no voters:
I20240502 14:00:04.604059 12734 raft_consensus.cc:2781] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:04.604113 12734 raft_consensus.cc:482] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:04.604159 12734 raft_consensus.cc:3037] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:04.604979 12734 raft_consensus.cc:504] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:04.605216 12734 leader_election.cc:288] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 election: Requested vote from peers 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:04.605602 12030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f585c28308464bb46aabdf0ad98a08" candidate_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a078666d30488f9c204d93e6394ef9"
I20240502 14:00:04.605710 12030 raft_consensus.cc:3037] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:04.606527 12030 raft_consensus.cc:2454] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73ce3d001cdf4d0c8e1487958b7694ea in term 1.
I20240502 14:00:04.606786 11703 leader_election.cc:302] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, 73ce3d001cdf4d0c8e1487958b7694ea; no voters:
I20240502 14:00:04.606940 12734 raft_consensus.cc:2781] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:04.607046 12734 raft_consensus.cc:686] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Becoming Leader. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Running, Role: LEADER
I20240502 14:00:04.607182 12734 consensus_queue.cc:237] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:04.608736 11623 catalog_manager.cc:5484] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129). New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } }
I20240502 14:00:04.832307 12789 raft_consensus.cc:482] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:04.832463 12789 raft_consensus.cc:504] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:04.832823 12789 leader_election.cc:288] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:04.833369 12030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd1b1872364411e86b810420c54034a" candidate_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a078666d30488f9c204d93e6394ef9" is_pre_election: true
I20240502 14:00:04.833504 12030 raft_consensus.cc:2454] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbf92f56d7144a22a6d482c73c7c7ea9 in term 0.
I20240502 14:00:04.833832 11834 leader_election.cc:302] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:04.834033 12789 raft_consensus.cc:2781] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:04.834092 12789 raft_consensus.cc:482] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:04.834120 12789 raft_consensus.cc:3037] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:04.834991 12789 raft_consensus.cc:504] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:04.835268 12789 leader_election.cc:288] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 election: Requested vote from peers 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:04.835704 12030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd1b1872364411e86b810420c54034a" candidate_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a078666d30488f9c204d93e6394ef9"
I20240502 14:00:04.835821 12030 raft_consensus.cc:3037] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:04.836544 12030 raft_consensus.cc:2454] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbf92f56d7144a22a6d482c73c7c7ea9 in term 1.
I20240502 14:00:04.836827 11834 leader_election.cc:302] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:04.837000 12789 raft_consensus.cc:2781] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:04.837081 12789 raft_consensus.cc:686] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Becoming Leader. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Running, Role: LEADER
I20240502 14:00:04.837203 12789 consensus_queue.cc:237] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:04.838837 11623 catalog_manager.cc:5484] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: HEALTHY } } }
I20240502 14:00:04.867444 12707 raft_consensus.cc:482] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:04.867599 12707 raft_consensus.cc:504] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:04.868016 12707 leader_election.cc:288] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 14:00:04.868577 11768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0230720456b49339caaa9d313b6ce64" candidate_uuid: "65a078666d30488f9c204d93e6394ef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" is_pre_election: true
I20240502 14:00:04.868750 11768 raft_consensus.cc:2454] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65a078666d30488f9c204d93e6394ef9 in term 0.
I20240502 14:00:04.869093 11965 leader_election.cc:302] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, 73ce3d001cdf4d0c8e1487958b7694ea; no voters:
I20240502 14:00:04.869272 12707 raft_consensus.cc:2781] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:04.869331 12707 raft_consensus.cc:482] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:04.869374 12707 raft_consensus.cc:3037] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:04.870218 12707 raft_consensus.cc:504] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:04.870505 12707 leader_election.cc:288] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [CANDIDATE]: Term 1 election: Requested vote from peers 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 14:00:04.870885 11768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0230720456b49339caaa9d313b6ce64" candidate_uuid: "65a078666d30488f9c204d93e6394ef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
I20240502 14:00:04.870988 11768 raft_consensus.cc:3037] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:04.871735 11768 raft_consensus.cc:2454] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65a078666d30488f9c204d93e6394ef9 in term 1.
I20240502 14:00:04.872017 11965 leader_election.cc:302] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, 73ce3d001cdf4d0c8e1487958b7694ea; no voters:
I20240502 14:00:04.872237 12707 raft_consensus.cc:2781] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:04.872457 12707 raft_consensus.cc:686] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Becoming Leader. State: Replica: 65a078666d30488f9c204d93e6394ef9, State: Running, Role: LEADER
I20240502 14:00:04.872599 12707 consensus_queue.cc:237] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:04.874202 11623 catalog_manager.cc:5484] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65a078666d30488f9c204d93e6394ef9 (127.8.86.131). New cstate: current_term: 1 leader_uuid: "65a078666d30488f9c204d93e6394ef9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: UNKNOWN } } }
W20240502 14:00:04.993342 12810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:04.993602 12810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:04.996752 12810 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 14:00:05.084116 12030 raft_consensus.cc:1261] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:05.084295 11768 raft_consensus.cc:1261] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:05.084717 12734 consensus_queue.cc:1035] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:05.084901 12707 consensus_queue.cc:1035] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:05.086238 12030 raft_consensus.cc:1261] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:05.086937 12789 consensus_queue.cc:1035] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 92.6376 ms
time per row: 0.0463188 ms
I20240502 14:00:05.131639 11626 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47428:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:05.136894 11878 tablet_service.cc:1467] Processing CreateTablet for tablet 236933482b6b41cf9e84a2eeb3b57638 (DEFAULT_TABLE table=rebalance_test_table_3 [id=22735e7421b84776ac7e9ae79a0aa1f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:05.136974 11748 tablet_service.cc:1467] Processing CreateTablet for tablet 236933482b6b41cf9e84a2eeb3b57638 (DEFAULT_TABLE table=rebalance_test_table_3 [id=22735e7421b84776ac7e9ae79a0aa1f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:05.137235 11748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 236933482b6b41cf9e84a2eeb3b57638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:05.137200 11877 tablet_service.cc:1467] Processing CreateTablet for tablet 1870fb0c459f49cab36812d16e30ddf5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=22735e7421b84776ac7e9ae79a0aa1f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:05.137297 11879 tablet_service.cc:1467] Processing CreateTablet for tablet 182302167a0541888cb99bcd66635125 (DEFAULT_TABLE table=rebalance_test_table_3 [id=22735e7421b84776ac7e9ae79a0aa1f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:05.137224 11878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 236933482b6b41cf9e84a2eeb3b57638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:05.137650 11877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1870fb0c459f49cab36812d16e30ddf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:05.137828 11879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182302167a0541888cb99bcd66635125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:05.138491 12008 tablet_service.cc:1467] Processing CreateTablet for tablet 1870fb0c459f49cab36812d16e30ddf5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=22735e7421b84776ac7e9ae79a0aa1f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:05.138769 12008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1870fb0c459f49cab36812d16e30ddf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:05.140414 11748 tablet_service.cc:1467] Processing CreateTablet for tablet 182302167a0541888cb99bcd66635125 (DEFAULT_TABLE table=rebalance_test_table_3 [id=22735e7421b84776ac7e9ae79a0aa1f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:05.140542 12862 tablet_bootstrap.cc:492] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea: Bootstrap starting.
I20240502 14:00:05.141013 12864 tablet_bootstrap.cc:492] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9: Bootstrap starting.
I20240502 14:00:05.141098 12863 tablet_bootstrap.cc:492] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9: Bootstrap starting.
I20240502 14:00:05.141389 11748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182302167a0541888cb99bcd66635125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:05.142325 12863 tablet_bootstrap.cc:654] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:05.142328 12862 tablet_bootstrap.cc:654] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:05.142572 12864 tablet_bootstrap.cc:654] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:05.143503 12863 tablet_bootstrap.cc:492] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9: No bootstrap required, opened a new log
I20240502 14:00:05.143601 12863 ts_tablet_manager.cc:1367] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:05.143746 12862 tablet_bootstrap.cc:492] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea: No bootstrap required, opened a new log
I20240502 14:00:05.143821 12862 ts_tablet_manager.cc:1367] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:05.144131 12863 raft_consensus.cc:348] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:05.144248 12863 raft_consensus.cc:374] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:05.144281 12863 raft_consensus.cc:729] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Initialized, Role: FOLLOWER
I20240502 14:00:05.144227 12862 raft_consensus.cc:348] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:05.144326 12862 raft_consensus.cc:374] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:05.144357 12862 raft_consensus.cc:729] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Initialized, Role: FOLLOWER
I20240502 14:00:05.144454 12864 tablet_bootstrap.cc:492] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9: No bootstrap required, opened a new log
I20240502 14:00:05.144415 12863 consensus_queue.cc:260] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:05.144479 12862 consensus_queue.cc:260] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:05.144526 12864 ts_tablet_manager.cc:1367] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:00:05.144706 12863 ts_tablet_manager.cc:1398] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:05.144745 12862 ts_tablet_manager.cc:1398] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:05.144865 12863 tablet_bootstrap.cc:492] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9: Bootstrap starting.
I20240502 14:00:05.144887 12862 tablet_bootstrap.cc:492] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea: Bootstrap starting.
I20240502 14:00:05.144963 12864 raft_consensus.cc:348] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:05.145052 12864 raft_consensus.cc:374] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:05.145082 12864 raft_consensus.cc:729] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a078666d30488f9c204d93e6394ef9, State: Initialized, Role: FOLLOWER
I20240502 14:00:05.145196 12864 consensus_queue.cc:260] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:05.145469 12864 ts_tablet_manager.cc:1398] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:05.145761 12863 tablet_bootstrap.cc:654] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:05.145787 12862 tablet_bootstrap.cc:654] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:05.147264 12863 tablet_bootstrap.cc:492] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9: No bootstrap required, opened a new log
I20240502 14:00:05.147352 12863 ts_tablet_manager.cc:1367] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:00:05.147743 12862 tablet_bootstrap.cc:492] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea: No bootstrap required, opened a new log
I20240502 14:00:05.147825 12863 raft_consensus.cc:348] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:05.147941 12863 raft_consensus.cc:374] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:05.147830 12862 ts_tablet_manager.cc:1367] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:00:05.147976 12863 raft_consensus.cc:729] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Initialized, Role: FOLLOWER
I20240502 14:00:05.148095 12863 consensus_queue.cc:260] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:05.148352 12863 ts_tablet_manager.cc:1398] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:05.148396 12862 raft_consensus.cc:348] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:05.148479 12862 raft_consensus.cc:374] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:05.148511 12862 raft_consensus.cc:729] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Initialized, Role: FOLLOWER
I20240502 14:00:05.148522 12863 tablet_bootstrap.cc:492] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9: Bootstrap starting.
I20240502 14:00:05.148654 12862 consensus_queue.cc:260] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:05.149004 12862 ts_tablet_manager.cc:1398] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:05.149392 12863 tablet_bootstrap.cc:654] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:05.150617 12863 tablet_bootstrap.cc:492] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9: No bootstrap required, opened a new log
I20240502 14:00:05.150704 12863 ts_tablet_manager.cc:1367] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:00:05.151100 12863 raft_consensus.cc:348] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:05.151211 12863 raft_consensus.cc:374] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:05.151255 12863 raft_consensus.cc:729] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Initialized, Role: FOLLOWER
I20240502 14:00:05.151371 12863 consensus_queue.cc:260] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:05.151640 12863 ts_tablet_manager.cc:1398] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:05.215740 12789 raft_consensus.cc:482] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:05.215871 12789 raft_consensus.cc:504] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:05.216236 12789 leader_election.cc:288] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 14:00:05.216820 11768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236933482b6b41cf9e84a2eeb3b57638" candidate_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" is_pre_election: true
I20240502 14:00:05.216985 11768 raft_consensus.cc:2454] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbf92f56d7144a22a6d482c73c7c7ea9 in term 0.
I20240502 14:00:05.217312 11834 leader_election.cc:302] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 73ce3d001cdf4d0c8e1487958b7694ea, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:05.217506 12789 raft_consensus.cc:2781] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:05.217563 12789 raft_consensus.cc:482] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:05.217602 12789 raft_consensus.cc:3037] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:05.218482 12789 raft_consensus.cc:504] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:05.218750 12789 leader_election.cc:288] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 election: Requested vote from peers 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 14:00:05.219134 11768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236933482b6b41cf9e84a2eeb3b57638" candidate_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
I20240502 14:00:05.219264 11768 raft_consensus.cc:3037] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:05.219950 11768 raft_consensus.cc:2454] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbf92f56d7144a22a6d482c73c7c7ea9 in term 1.
I20240502 14:00:05.220304 11834 leader_election.cc:302] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 73ce3d001cdf4d0c8e1487958b7694ea, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:05.220597 12789 raft_consensus.cc:2781] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:05.220680 12789 raft_consensus.cc:686] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Becoming Leader. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Running, Role: LEADER
I20240502 14:00:05.220827 12789 consensus_queue.cc:237] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:05.222604 11622 catalog_manager.cc:5484] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:05.314893 12707 raft_consensus.cc:482] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:05.315049 12707 raft_consensus.cc:504] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:05.315400 12707 leader_election.cc:288] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377)
I20240502 14:00:05.315954 11898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1870fb0c459f49cab36812d16e30ddf5" candidate_uuid: "65a078666d30488f9c204d93e6394ef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" is_pre_election: true
I20240502 14:00:05.316084 11898 raft_consensus.cc:2454] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65a078666d30488f9c204d93e6394ef9 in term 0.
I20240502 14:00:05.316454 11966 leader_election.cc:302] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:05.316654 12707 raft_consensus.cc:2781] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:05.316709 12707 raft_consensus.cc:482] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:05.316759 12707 raft_consensus.cc:3037] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:05.317641 12707 raft_consensus.cc:504] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:05.317956 12707 leader_election.cc:288] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [CANDIDATE]: Term 1 election: Requested vote from peers cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377)
I20240502 14:00:05.318444 11898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1870fb0c459f49cab36812d16e30ddf5" candidate_uuid: "65a078666d30488f9c204d93e6394ef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9"
I20240502 14:00:05.318552 11898 raft_consensus.cc:3037] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:05.319427 11898 raft_consensus.cc:2454] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65a078666d30488f9c204d93e6394ef9 in term 1.
I20240502 14:00:05.319526 12789 raft_consensus.cc:482] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:05.319635 12789 raft_consensus.cc:504] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:05.319761 11966 leader_election.cc:302] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:05.319938 12707 raft_consensus.cc:2781] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:05.319986 12789 leader_election.cc:288] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 14:00:05.320027 12707 raft_consensus.cc:686] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Becoming Leader. State: Replica: 65a078666d30488f9c204d93e6394ef9, State: Running, Role: LEADER
I20240502 14:00:05.320171 12707 consensus_queue.cc:237] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:05.320544 11768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182302167a0541888cb99bcd66635125" candidate_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" is_pre_election: true
I20240502 14:00:05.320732 11768 raft_consensus.cc:2454] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbf92f56d7144a22a6d482c73c7c7ea9 in term 0.
I20240502 14:00:05.321149 11834 leader_election.cc:302] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 73ce3d001cdf4d0c8e1487958b7694ea, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:05.321316 12789 raft_consensus.cc:2781] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:05.321390 12789 raft_consensus.cc:482] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:05.321439 12789 raft_consensus.cc:3037] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:05.322075 11622 catalog_manager.cc:5484] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65a078666d30488f9c204d93e6394ef9 (127.8.86.131). New cstate: current_term: 1 leader_uuid: "65a078666d30488f9c204d93e6394ef9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: HEALTHY } } }
I20240502 14:00:05.322161 12789 raft_consensus.cc:504] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:05.322454 12789 leader_election.cc:288] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 election: Requested vote from peers 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 14:00:05.322849 11768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182302167a0541888cb99bcd66635125" candidate_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
I20240502 14:00:05.322958 11768 raft_consensus.cc:3037] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:05.323765 11768 raft_consensus.cc:2454] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbf92f56d7144a22a6d482c73c7c7ea9 in term 1.
I20240502 14:00:05.324066 11834 leader_election.cc:302] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 73ce3d001cdf4d0c8e1487958b7694ea, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:05.324223 12789 raft_consensus.cc:2781] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:05.324326 12789 raft_consensus.cc:686] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Becoming Leader. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Running, Role: LEADER
I20240502 14:00:05.324465 12789 consensus_queue.cc:237] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:05.325979 11622 catalog_manager.cc:5484] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: HEALTHY } } }
W20240502 14:00:05.441126 12874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:05.441401 12874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:05.444520 12874 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 14:00:05.529568 11768 raft_consensus.cc:1261] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:05.530181 11898 raft_consensus.cc:1261] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:05.530802 12707 consensus_queue.cc:1035] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:05.531373 12789 consensus_queue.cc:1035] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:05.533241 11768 raft_consensus.cc:1261] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:05.533707 12740 consensus_queue.cc:1035] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 89.6994 ms
time per row: 0.0448497 ms
I20240502 14:00:05.577780 11623 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47428:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:05.583536 11748 tablet_service.cc:1467] Processing CreateTablet for tablet 259e249430fa41a2bc2528b65d022b67 (DEFAULT_TABLE table=rebalance_test_table_4 [id=83b2409333254a509079ef88e0c156f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:05.583673 12008 tablet_service.cc:1467] Processing CreateTablet for tablet 259e249430fa41a2bc2528b65d022b67 (DEFAULT_TABLE table=rebalance_test_table_4 [id=83b2409333254a509079ef88e0c156f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:05.583911 11748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 259e249430fa41a2bc2528b65d022b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:05.584012 12008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 259e249430fa41a2bc2528b65d022b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:05.583997 11877 tablet_service.cc:1467] Processing CreateTablet for tablet dc89c51bfacd43d7a7cca53bc5624438 (DEFAULT_TABLE table=rebalance_test_table_4 [id=83b2409333254a509079ef88e0c156f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:05.584275 11877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc89c51bfacd43d7a7cca53bc5624438. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:05.584380 11747 tablet_service.cc:1467] Processing CreateTablet for tablet b9bc1c84cb144965a21bdc40f099416d (DEFAULT_TABLE table=rebalance_test_table_4 [id=83b2409333254a509079ef88e0c156f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:05.584596 11747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9bc1c84cb144965a21bdc40f099416d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:05.586367 12863 tablet_bootstrap.cc:492] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9: Bootstrap starting.
I20240502 14:00:05.586424 12864 tablet_bootstrap.cc:492] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9: Bootstrap starting.
I20240502 14:00:05.586469 12862 tablet_bootstrap.cc:492] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea: Bootstrap starting.
I20240502 14:00:05.587250 12864 tablet_bootstrap.cc:654] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:05.587252 12862 tablet_bootstrap.cc:654] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:05.587252 12863 tablet_bootstrap.cc:654] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:05.583701 12009 tablet_service.cc:1467] Processing CreateTablet for tablet dc89c51bfacd43d7a7cca53bc5624438 (DEFAULT_TABLE table=rebalance_test_table_4 [id=83b2409333254a509079ef88e0c156f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:05.587961 12009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc89c51bfacd43d7a7cca53bc5624438. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:05.588316 12008 tablet_service.cc:1467] Processing CreateTablet for tablet b9bc1c84cb144965a21bdc40f099416d (DEFAULT_TABLE table=rebalance_test_table_4 [id=83b2409333254a509079ef88e0c156f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:05.588537 12008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9bc1c84cb144965a21bdc40f099416d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:05.589635 12864 tablet_bootstrap.cc:492] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9: No bootstrap required, opened a new log
I20240502 14:00:05.589717 12864 ts_tablet_manager.cc:1367] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:00:05.590219 12862 tablet_bootstrap.cc:492] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea: No bootstrap required, opened a new log
I20240502 14:00:05.590178 12864 raft_consensus.cc:348] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:05.590328 12862 ts_tablet_manager.cc:1367] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:00:05.590334 12864 raft_consensus.cc:374] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:05.590370 12864 raft_consensus.cc:729] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a078666d30488f9c204d93e6394ef9, State: Initialized, Role: FOLLOWER
I20240502 14:00:05.590493 12864 consensus_queue.cc:260] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:05.590811 12864 ts_tablet_manager.cc:1398] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:05.590768 12862 raft_consensus.cc:348] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:05.590865 12862 raft_consensus.cc:374] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:05.590906 12862 raft_consensus.cc:729] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Initialized, Role: FOLLOWER
I20240502 14:00:05.590996 12864 tablet_bootstrap.cc:492] T dc89c51bfacd43d7a7cca53bc5624438 P 65a078666d30488f9c204d93e6394ef9: Bootstrap starting.
I20240502 14:00:05.591041 12862 consensus_queue.cc:260] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:05.591307 12862 ts_tablet_manager.cc:1398] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:05.591475 12862 tablet_bootstrap.cc:492] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea: Bootstrap starting.
I20240502 14:00:05.591873 12864 tablet_bootstrap.cc:654] T dc89c51bfacd43d7a7cca53bc5624438 P 65a078666d30488f9c204d93e6394ef9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:05.592257 12862 tablet_bootstrap.cc:654] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:05.593016 12864 tablet_bootstrap.cc:492] T dc89c51bfacd43d7a7cca53bc5624438 P 65a078666d30488f9c204d93e6394ef9: No bootstrap required, opened a new log
I20240502 14:00:05.593099 12864 ts_tablet_manager.cc:1367] T dc89c51bfacd43d7a7cca53bc5624438 P 65a078666d30488f9c204d93e6394ef9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:05.593462 12864 raft_consensus.cc:348] T dc89c51bfacd43d7a7cca53bc5624438 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:05.593555 12864 raft_consensus.cc:374] T dc89c51bfacd43d7a7cca53bc5624438 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:05.593590 12864 raft_consensus.cc:729] T dc89c51bfacd43d7a7cca53bc5624438 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a078666d30488f9c204d93e6394ef9, State: Initialized, Role: FOLLOWER
I20240502 14:00:05.593731 12864 consensus_queue.cc:260] T dc89c51bfacd43d7a7cca53bc5624438 P 65a078666d30488f9c204d93e6394ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:05.594027 12863 tablet_bootstrap.cc:492] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9: No bootstrap required, opened a new log
I20240502 14:00:05.594086 12864 ts_tablet_manager.cc:1398] T dc89c51bfacd43d7a7cca53bc5624438 P 65a078666d30488f9c204d93e6394ef9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:05.594137 12863 ts_tablet_manager.cc:1367] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240502 14:00:05.594256 12864 tablet_bootstrap.cc:492] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9: Bootstrap starting.
I20240502 14:00:05.594441 12862 tablet_bootstrap.cc:492] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea: No bootstrap required, opened a new log
I20240502 14:00:05.594513 12862 ts_tablet_manager.cc:1367] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:05.594626 12863 raft_consensus.cc:348] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:05.594728 12863 raft_consensus.cc:374] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:05.594759 12863 raft_consensus.cc:729] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Initialized, Role: FOLLOWER
I20240502 14:00:05.594875 12863 consensus_queue.cc:260] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:05.595021 12862 raft_consensus.cc:348] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:05.595118 12862 raft_consensus.cc:374] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:05.595149 12862 raft_consensus.cc:729] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Initialized, Role: FOLLOWER
I20240502 14:00:05.595153 12863 ts_tablet_manager.cc:1398] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:05.595270 12862 consensus_queue.cc:260] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:05.595515 12864 tablet_bootstrap.cc:654] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:05.595562 12862 ts_tablet_manager.cc:1398] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:05.597008 12864 tablet_bootstrap.cc:492] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9: No bootstrap required, opened a new log
I20240502 14:00:05.597095 12864 ts_tablet_manager.cc:1367] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:05.597476 12864 raft_consensus.cc:348] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:05.597568 12864 raft_consensus.cc:374] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:05.597609 12864 raft_consensus.cc:729] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a078666d30488f9c204d93e6394ef9, State: Initialized, Role: FOLLOWER
I20240502 14:00:05.597764 12864 consensus_queue.cc:260] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:05.598045 12864 ts_tablet_manager.cc:1398] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:05.734895 12906 raft_consensus.cc:482] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:05.735041 12906 raft_consensus.cc:504] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:05.735455 12906 leader_election.cc:288] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:05.735983 12029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc89c51bfacd43d7a7cca53bc5624438" candidate_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a078666d30488f9c204d93e6394ef9" is_pre_election: true
I20240502 14:00:05.736129 12029 raft_consensus.cc:2454] T dc89c51bfacd43d7a7cca53bc5624438 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbf92f56d7144a22a6d482c73c7c7ea9 in term 0.
I20240502 14:00:05.736452 11834 leader_election.cc:302] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:05.736644 12906 raft_consensus.cc:2781] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:05.736703 12906 raft_consensus.cc:482] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:05.736732 12906 raft_consensus.cc:3037] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:05.737556 12906 raft_consensus.cc:504] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:05.737828 12906 leader_election.cc:288] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 election: Requested vote from peers 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:05.738220 12029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc89c51bfacd43d7a7cca53bc5624438" candidate_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a078666d30488f9c204d93e6394ef9"
I20240502 14:00:05.738407 12029 raft_consensus.cc:3037] T dc89c51bfacd43d7a7cca53bc5624438 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:05.739212 12029 raft_consensus.cc:2454] T dc89c51bfacd43d7a7cca53bc5624438 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbf92f56d7144a22a6d482c73c7c7ea9 in term 1.
I20240502 14:00:05.739528 11834 leader_election.cc:302] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:05.739720 12906 raft_consensus.cc:2781] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:05.739804 12906 raft_consensus.cc:686] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Becoming Leader. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Running, Role: LEADER
I20240502 14:00:05.739941 12906 consensus_queue.cc:237] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } }
I20240502 14:00:05.741587 11624 catalog_manager.cc:5484] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: HEALTHY } } }
I20240502 14:00:05.776682 12861 raft_consensus.cc:482] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:05.776820 12861 raft_consensus.cc:504] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:05.777312 12861 leader_election.cc:288] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:05.777802 12029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "259e249430fa41a2bc2528b65d022b67" candidate_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a078666d30488f9c204d93e6394ef9" is_pre_election: true
I20240502 14:00:05.777943 12029 raft_consensus.cc:2454] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73ce3d001cdf4d0c8e1487958b7694ea in term 0.
I20240502 14:00:05.778278 11703 leader_election.cc:302] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, 73ce3d001cdf4d0c8e1487958b7694ea; no voters:
I20240502 14:00:05.778471 12861 raft_consensus.cc:2781] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:05.778548 12861 raft_consensus.cc:482] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:05.778584 12861 raft_consensus.cc:3037] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:05.779438 12861 raft_consensus.cc:504] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:05.779712 12861 leader_election.cc:288] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 election: Requested vote from peers 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:05.780050 12029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "259e249430fa41a2bc2528b65d022b67" candidate_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a078666d30488f9c204d93e6394ef9"
I20240502 14:00:05.780155 12029 raft_consensus.cc:3037] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:05.781009 12029 raft_consensus.cc:2454] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73ce3d001cdf4d0c8e1487958b7694ea in term 1.
I20240502 14:00:05.781304 11703 leader_election.cc:302] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, 73ce3d001cdf4d0c8e1487958b7694ea; no voters:
I20240502 14:00:05.781446 12861 raft_consensus.cc:2781] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:05.781615 12861 raft_consensus.cc:686] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Becoming Leader. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Running, Role: LEADER
I20240502 14:00:05.781767 12861 consensus_queue.cc:237] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } }
I20240502 14:00:05.783290 11624 catalog_manager.cc:5484] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129). New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:05.791765 12861 raft_consensus.cc:482] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:05.791873 12861 raft_consensus.cc:504] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:05.792143 12861 leader_election.cc:288] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:05.792564 12029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9bc1c84cb144965a21bdc40f099416d" candidate_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a078666d30488f9c204d93e6394ef9" is_pre_election: true
I20240502 14:00:05.792682 12029 raft_consensus.cc:2454] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73ce3d001cdf4d0c8e1487958b7694ea in term 0.
I20240502 14:00:05.792932 11703 leader_election.cc:302] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, 73ce3d001cdf4d0c8e1487958b7694ea; no voters:
I20240502 14:00:05.793081 12861 raft_consensus.cc:2781] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:05.793138 12861 raft_consensus.cc:482] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:05.793170 12861 raft_consensus.cc:3037] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:05.793895 12861 raft_consensus.cc:504] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:05.794123 12861 leader_election.cc:288] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 election: Requested vote from peers 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:05.794507 12029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9bc1c84cb144965a21bdc40f099416d" candidate_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a078666d30488f9c204d93e6394ef9"
I20240502 14:00:05.794624 12029 raft_consensus.cc:3037] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:05.795292 12029 raft_consensus.cc:2454] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73ce3d001cdf4d0c8e1487958b7694ea in term 1.
I20240502 14:00:05.795555 11703 leader_election.cc:302] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65a078666d30488f9c204d93e6394ef9, 73ce3d001cdf4d0c8e1487958b7694ea; no voters:
I20240502 14:00:05.795703 12861 raft_consensus.cc:2781] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:05.795801 12861 raft_consensus.cc:686] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Becoming Leader. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Running, Role: LEADER
I20240502 14:00:05.795945 12861 consensus_queue.cc:237] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } }
I20240502 14:00:05.798135 11624 catalog_manager.cc:5484] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129). New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } }
W20240502 14:00:05.935639 12919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:05.935894 12919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:05.938910 12919 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 14:00:06.037802 12029 raft_consensus.cc:1261] T dc89c51bfacd43d7a7cca53bc5624438 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:06.038249 12906 consensus_queue.cc:1035] T dc89c51bfacd43d7a7cca53bc5624438 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:06.038787 12030 raft_consensus.cc:1261] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:06.039326 12861 consensus_queue.cc:1035] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:06.039659 12029 raft_consensus.cc:1261] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:06.040464 12916 consensus_queue.cc:1035] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 104.944 ms
time per row: 0.0524721 ms
I20240502 14:00:06.085906 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.132:42709
--local_ip_for_outbound_sockets=127.8.86.132
--tserver_master_addrs=127.8.86.190:44201
--webserver_port=45443
--webserver_interface=127.8.86.132
--builtin_ntp_servers=127.8.86.148:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:00:06.195079 12958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:06.195317 12958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:06.195360 12958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:06.198618 12958 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:00:06.198695 12958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:06.198770 12958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.132
I20240502 14:00:06.202416 12958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.132:42709
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.86.132
--webserver_port=45443
--tserver_master_addrs=127.8.86.190:44201
--never_fsync=true
--heap_profile_path=/tmp/kudu.12958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.132
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:06.203338 12958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:06.204455 12958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 14:00:06.213238 12958 server_base.cc:1034] running on GCE node
W20240502 14:00:06.213716 12963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:06.214432 12964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:06.215071 12966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:06.215497 12958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:06.216037 12958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:06.217183 12958 hybrid_clock.cc:648] HybridClock initialized: now 1714658406217160 us; error 45 us; skew 500 ppm
I20240502 14:00:06.219105 12958 webserver.cc:460] Webserver started at http://127.8.86.132:45443/ using document root <none> and password file <none>
I20240502 14:00:06.219728 12958 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:06.219810 12958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:06.223220 12958 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20240502 14:00:06.225026 12972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:06.226114 12958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:06.226217 12958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4f6f2d2b1569415cbbbea2466dc28e28"
format_stamp: "Formatted at 2024-05-02 13:59:56 on dist-test-slave-h6v0"
I20240502 14:00:06.226663 12958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:06.240271 12958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:06.240928 12958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:06.241089 12958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:06.241607 12958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:06.242494 12958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:06.242545 12958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:06.242602 12958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:06.242630 12958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:06.251583 12958 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.132:42709
I20240502 14:00:06.251616 13085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.132:42709 every 8 connection(s)
I20240502 14:00:06.252683 12958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:00:06.261970 13086 heartbeater.cc:347] Connected to a master server at 127.8.86.190:44201
I20240502 14:00:06.262197 13086 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:06.262555 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 12958
I20240502 14:00:06.262856 13086 heartbeater.cc:511] Master 127.8.86.190:44201 requested a full tablet report, sending...
I20240502 14:00:06.262985 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.133:44919
--local_ip_for_outbound_sockets=127.8.86.133
--tserver_master_addrs=127.8.86.190:44201
--webserver_port=36609
--webserver_interface=127.8.86.133
--builtin_ntp_servers=127.8.86.148:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:00:06.264088 11624 ts_manager.cc:194] Re-registered known tserver with Master: 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132:42709)
I20240502 14:00:06.264817 11624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.132:46177
W20240502 14:00:06.375509 13089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:06.375764 13089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:06.375841 13089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:06.379024 13089 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:00:06.379112 13089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:06.379204 13089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.133
I20240502 14:00:06.382936 13089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.133:44919
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.86.133
--webserver_port=36609
--tserver_master_addrs=127.8.86.190:44201
--never_fsync=true
--heap_profile_path=/tmp/kudu.13089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.133
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:06.383862 13089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:06.385000 13089 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:06.393534 13094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:06.394192 13095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:06.394830 13097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:06.395749 13089 server_base.cc:1034] running on GCE node
I20240502 14:00:06.396200 13089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:06.396827 13089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:06.397974 13089 hybrid_clock.cc:648] HybridClock initialized: now 1714658406397957 us; error 34 us; skew 500 ppm
I20240502 14:00:06.399945 13089 webserver.cc:460] Webserver started at http://127.8.86.133:36609/ using document root <none> and password file <none>
I20240502 14:00:06.400494 13089 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:06.400566 13089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:06.403861 13089 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20240502 14:00:06.405738 13103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:06.406750 13089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:06.406848 13089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "5fbb448689c949d98dec8e37cf5c3478"
format_stamp: "Formatted at 2024-05-02 13:59:56 on dist-test-slave-h6v0"
I20240502 14:00:06.407266 13089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:06.423915 13089 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:06.424561 13089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:06.424722 13089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:06.425262 13089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:06.426138 13089 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:06.426185 13089 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:06.426226 13089 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:06.426254 13089 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:06.434574 13089 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.133:44919
I20240502 14:00:06.434612 13217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.133:44919 every 8 connection(s)
I20240502 14:00:06.435570 13089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:00:06.439122 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 13089
I20240502 14:00:06.439502 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.134:36541
--local_ip_for_outbound_sockets=127.8.86.134
--tserver_master_addrs=127.8.86.190:44201
--webserver_port=33731
--webserver_interface=127.8.86.134
--builtin_ntp_servers=127.8.86.148:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:00:06.444298 13218 heartbeater.cc:347] Connected to a master server at 127.8.86.190:44201
I20240502 14:00:06.444562 13218 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:06.445282 13218 heartbeater.cc:511] Master 127.8.86.190:44201 requested a full tablet report, sending...
I20240502 14:00:06.446482 11624 ts_manager.cc:194] Re-registered known tserver with Master: 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133:44919)
I20240502 14:00:06.447023 11624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.133:57439
W20240502 14:00:06.558197 13221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:06.558441 13221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:06.558501 13221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:06.561565 13221 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:00:06.561625 13221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:06.561712 13221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.134
I20240502 14:00:06.565472 13221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.134:36541
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.8.86.134
--webserver_port=33731
--tserver_master_addrs=127.8.86.190:44201
--never_fsync=true
--heap_profile_path=/tmp/kudu.13221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.134
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:06.566386 13221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:06.567587 13221 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:06.575954 13229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:06.576647 13227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:06.577280 13221 server_base.cc:1034] running on GCE node
W20240502 14:00:06.578055 13226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:06.578439 13221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:06.578998 13221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:06.580153 13221 hybrid_clock.cc:648] HybridClock initialized: now 1714658406580143 us; error 22 us; skew 500 ppm
I20240502 14:00:06.585393 13221 webserver.cc:460] Webserver started at http://127.8.86.134:33731/ using document root <none> and password file <none>
I20240502 14:00:06.585958 13221 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:06.586030 13221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:06.589426 13221 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:06.591269 13235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:06.592228 13221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:06.592330 13221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "a36d5a88b1a34006bd3d040f5a715766"
format_stamp: "Formatted at 2024-05-02 13:59:56 on dist-test-slave-h6v0"
I20240502 14:00:06.592737 13221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:06.606122 13221 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:06.606701 13221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:06.606858 13221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:06.607332 13221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:06.608181 13221 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:06.608227 13221 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:06.608271 13221 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:06.608302 13221 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:06.616649 13221 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.134:36541
I20240502 14:00:06.616675 13348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.134:36541 every 8 connection(s)
I20240502 14:00:06.617604 13221 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:00:06.625934 13349 heartbeater.cc:347] Connected to a master server at 127.8.86.190:44201
I20240502 14:00:06.626152 13349 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:06.626482 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 13221
I20240502 14:00:06.626683 13349 heartbeater.cc:511] Master 127.8.86.190:44201 requested a full tablet report, sending...
I20240502 14:00:06.626850 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.135:34019
--local_ip_for_outbound_sockets=127.8.86.135
--tserver_master_addrs=127.8.86.190:44201
--webserver_port=38241
--webserver_interface=127.8.86.135
--builtin_ntp_servers=127.8.86.148:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:00:06.627682 11624 ts_manager.cc:194] Re-registered known tserver with Master: a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134:36541)
I20240502 14:00:06.628223 11624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.134:35297
W20240502 14:00:06.741770 13352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:06.742050 13352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:06.742136 13352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:06.745286 13352 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:00:06.745374 13352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:06.745481 13352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.135
I20240502 14:00:06.749259 13352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.135:34019
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.8.86.135
--webserver_port=38241
--tserver_master_addrs=127.8.86.190:44201
--never_fsync=true
--heap_profile_path=/tmp/kudu.13352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.135
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:06.750252 13352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:06.751459 13352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:06.760260 13360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:06.760777 13358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:06.761543 13357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:06.762336 13352 server_base.cc:1034] running on GCE node
I20240502 14:00:06.762820 13352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:06.763454 13352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:06.764606 13352 hybrid_clock.cc:648] HybridClock initialized: now 1714658406764572 us; error 47 us; skew 500 ppm
I20240502 14:00:06.766724 13352 webserver.cc:460] Webserver started at http://127.8.86.135:38241/ using document root <none> and password file <none>
I20240502 14:00:06.767320 13352 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:06.767408 13352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:06.771039 13352 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:00:06.773085 13366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:06.774168 13352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:06.774315 13352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "7553d1bab6f642ac8395c928da1c58d7"
format_stamp: "Formatted at 2024-05-02 13:59:56 on dist-test-slave-h6v0"
I20240502 14:00:06.774786 13352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:06.809979 13352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:06.810748 13352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:06.810896 13352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:06.811437 13352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:06.812420 13352 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:06.812474 13352 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:06.812518 13352 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:06.812553 13352 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:06.821472 13352 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.135:34019
I20240502 14:00:06.821516 13479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.135:34019 every 8 connection(s)
I20240502 14:00:06.822676 13352 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:00:06.824371 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 13352
I20240502 14:00:06.824553 8538 test_util.cc:268] Using random seed: 88078952
I20240502 14:00:06.832221 13480 heartbeater.cc:347] Connected to a master server at 127.8.86.190:44201
I20240502 14:00:06.832466 13480 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:06.833066 13480 heartbeater.cc:511] Master 127.8.86.190:44201 requested a full tablet report, sending...
I20240502 14:00:06.834049 11622 ts_manager.cc:194] Re-registered known tserver with Master: 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135:34019)
I20240502 14:00:06.834488 8538 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240502 14:00:06.834703 8538 test_util.cc:268] Using random seed: 88089101
I20240502 14:00:06.834695 11622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.135:51045
I20240502 14:00:06.844328 8538 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240502 14:00:06.844492 8538 test_util.cc:268] Using random seed: 88098891
I20240502 14:00:06.856171 8538 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240502 14:00:06.856343 8538 test_util.cc:268] Using random seed: 88110740
I20240502 14:00:06.871026 8538 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240502 14:00:06.871205 8538 test_util.cc:268] Using random seed: 88125602
I20240502 14:00:06.910053 8538 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240502 14:00:07.267052 13086 heartbeater.cc:503] Master 127.8.86.190:44201 was elected leader, sending a full tablet report...
I20240502 14:00:07.309078 13020 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:07.323041 12008 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:07.323767 11747 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:07.333663 13151 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:07.345928 13283 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:07.391372 11877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:07.393676 13414 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:07.449394 13218 heartbeater.cc:503] Master 127.8.86.190:44201 was elected leader, sending a full tablet report...
I20240502 14:00:07.630018 13349 heartbeater.cc:503] Master 127.8.86.190:44201 was elected leader, sending a full tablet report...
I20240502 14:00:07.650910 13020 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:07.664103 13283 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:07.677145 13151 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:07.679214 13414 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:07.693799 11877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:07.702622 11747 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:07.719627 12008 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:07.788959 11899 consensus_queue.cc:237] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.36, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } }
I20240502 14:00:07.791016 12029 raft_consensus.cc:1261] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20240502 14:00:07.795217 13572 consensus_queue.cc:1035] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 35, Time since last communication: 0.000s
I20240502 14:00:07.801235 13572 raft_consensus.cc:2932] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Committing config change with OpId 1.37: config changed from index -1 to 37, NON_VOTER 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } } }
I20240502 14:00:07.802156 12030 raft_consensus.cc:2932] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index -1 to 37, NON_VOTER 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } } }
W20240502 14:00:07.802848 11834 consensus_peers.cc:485] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 -> Peer 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133:44919): Couldn't send request to peer 5fbb448689c949d98dec8e37cf5c3478. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cd1b1872364411e86b810420c54034a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:07.815392 11624 catalog_manager.cc:5484] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9 reported cstate change: config changed from index -1 to 37, NON_VOTER 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } } }
I20240502 14:00:07.837224 13480 heartbeater.cc:503] Master 127.8.86.190:44201 was elected leader, sending a full tablet report...
I20240502 14:00:07.865449 11768 consensus_queue.cc:237] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 2.39, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } }
I20240502 14:00:07.868655 11899 raft_consensus.cc:1261] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 39. Preceding OpId from leader: term: 2 index: 40. (index mismatch)
I20240502 14:00:07.869792 13569 consensus_queue.cc:1035] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20240502 14:00:07.871762 12861 raft_consensus.cc:2932] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 LEADER]: Committing config change with OpId 2.40: config changed from index -1 to 40, NON_VOTER 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } } }
I20240502 14:00:07.874182 11899 raft_consensus.cc:2932] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Committing config change with OpId 2.40: config changed from index -1 to 40, NON_VOTER 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } } }
I20240502 14:00:07.888841 11619 catalog_manager.cc:5484] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: config changed from index -1 to 40, NON_VOTER 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) added. New cstate: current_term: 2 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:00:07.900673 11704 consensus_peers.cc:485] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea -> Peer 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132:42709): Couldn't send request to peer 4f6f2d2b1569415cbbbea2466dc28e28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9a6f7fca259430dac2ca17a8b180b2a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:07.926282 11767 consensus_queue.cc:237] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.34, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } }
I20240502 14:00:07.931001 12028 raft_consensus.cc:1261] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
W20240502 14:00:07.933907 11701 consensus_peers.cc:485] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea -> Peer a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134:36541): Couldn't send request to peer a36d5a88b1a34006bd3d040f5a715766. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c951304cdaa54040a81e51d076fdeaaf. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:07.940802 13566 consensus_queue.cc:1035] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240502 14:00:07.945837 13566 raft_consensus.cc:2932] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Committing config change with OpId 1.35: config changed from index -1 to 35, NON_VOTER a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } } }
I20240502 14:00:07.949935 12029 raft_consensus.cc:2932] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index -1 to 35, NON_VOTER a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } } }
I20240502 14:00:07.967844 11619 catalog_manager.cc:5484] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: config changed from index -1 to 35, NON_VOTER a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:08.003078 12027 consensus_queue.cc:237] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } }
I20240502 14:00:08.004316 11765 raft_consensus.cc:1261] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240502 14:00:08.005072 13541 consensus_queue.cc:1035] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240502 14:00:08.006619 13518 raft_consensus.cc:2932] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index -1 to 32, NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } } }
I20240502 14:00:08.007601 11765 raft_consensus.cc:2932] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index -1 to 32, NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } } }
W20240502 14:00:08.034610 11966 consensus_peers.cc:485] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 -> Peer 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135:34019): Couldn't send request to peer 7553d1bab6f642ac8395c928da1c58d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0230720456b49339caaa9d313b6ce64. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:08.095983 11619 catalog_manager.cc:5484] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 reported cstate change: config changed from index -1 to 32, NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "65a078666d30488f9c204d93e6394ef9" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:08.131793 11898 consensus_queue.cc:237] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } }
I20240502 14:00:08.136433 11768 raft_consensus.cc:1261] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:00:08.137971 11834 consensus_peers.cc:485] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 -> Peer 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133:44919): Couldn't send request to peer 5fbb448689c949d98dec8e37cf5c3478. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03409b5f413f499c919f2c6a5a4126f7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:08.138779 13572 consensus_queue.cc:1035] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:00:08.149279 13539 raft_consensus.cc:2932] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } } }
I20240502 14:00:08.151427 11768 raft_consensus.cc:2932] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } } }
I20240502 14:00:08.194953 11624 catalog_manager.cc:5484] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: config changed from index -1 to 2, NON_VOTER 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:08.224864 11768 consensus_queue.cc:237] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 1.49, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } }
I20240502 14:00:08.231659 12029 raft_consensus.cc:1261] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 49. Preceding OpId from leader: term: 1 index: 50. (index mismatch)
W20240502 14:00:08.231717 11704 consensus_peers.cc:485] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea -> Peer 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132:42709): Couldn't send request to peer 4f6f2d2b1569415cbbbea2466dc28e28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11f585c28308464bb46aabdf0ad98a08. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:08.232358 13569 consensus_queue.cc:1035] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20240502 14:00:08.234807 13559 raft_consensus.cc:2932] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Committing config change with OpId 1.50: config changed from index -1 to 50, NON_VOTER 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } } }
I20240502 14:00:08.235267 12028 raft_consensus.cc:2932] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Committing config change with OpId 1.50: config changed from index -1 to 50, NON_VOTER 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } } }
I20240502 14:00:08.295253 11624 catalog_manager.cc:5484] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: config changed from index -1 to 50, NON_VOTER 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) added. New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:08.311180 11767 consensus_queue.cc:237] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 1.45, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } }
I20240502 14:00:08.312700 12029 raft_consensus.cc:1261] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 45. Preceding OpId from leader: term: 1 index: 46. (index mismatch)
W20240502 14:00:08.313279 11701 consensus_peers.cc:485] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea -> Peer a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134:36541): Couldn't send request to peer a36d5a88b1a34006bd3d040f5a715766. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 259e249430fa41a2bc2528b65d022b67. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:08.314309 13700 consensus_queue.cc:1035] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20240502 14:00:08.316231 13699 raft_consensus.cc:2932] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } } }
I20240502 14:00:08.316671 12029 raft_consensus.cc:2932] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } } }
I20240502 14:00:08.376232 13706 ts_tablet_manager.cc:902] T 7cd1b1872364411e86b810420c54034a P 5fbb448689c949d98dec8e37cf5c3478: Initiating tablet copy from peer cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377)
I20240502 14:00:08.377568 13706 tablet_copy_client.cc:308] T 7cd1b1872364411e86b810420c54034a P 5fbb448689c949d98dec8e37cf5c3478: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:44377
I20240502 14:00:08.388999 11919 tablet_copy_service.cc:140] P cbf92f56d7144a22a6d482c73c7c7ea9: Received BeginTabletCopySession request for tablet 7cd1b1872364411e86b810420c54034a from peer 5fbb448689c949d98dec8e37cf5c3478 ({username='slave'} at 127.8.86.133:34065)
I20240502 14:00:08.389169 11919 tablet_copy_service.cc:161] P cbf92f56d7144a22a6d482c73c7c7ea9: Beginning new tablet copy session on tablet 7cd1b1872364411e86b810420c54034a from peer 5fbb448689c949d98dec8e37cf5c3478 at {username='slave'} at 127.8.86.133:34065: session id = 5fbb448689c949d98dec8e37cf5c3478-7cd1b1872364411e86b810420c54034a
I20240502 14:00:08.390746 11919 tablet_copy_source_session.cc:205] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:08.394683 11619 catalog_manager.cc:5484] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: config changed from index -1 to 46, NON_VOTER a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:08.399920 13706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cd1b1872364411e86b810420c54034a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:08.419530 13706 tablet_copy_client.cc:783] T 7cd1b1872364411e86b810420c54034a P 5fbb448689c949d98dec8e37cf5c3478: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:08.419941 13706 tablet_copy_client.cc:647] T 7cd1b1872364411e86b810420c54034a P 5fbb448689c949d98dec8e37cf5c3478: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:08.423401 13706 tablet_copy_client.cc:523] T 7cd1b1872364411e86b810420c54034a P 5fbb448689c949d98dec8e37cf5c3478: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:08.427747 12029 consensus_queue.cc:237] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 1.63, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } }
W20240502 14:00:08.431113 11966 consensus_peers.cc:485] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 -> Peer 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135:34019): Couldn't send request to peer 7553d1bab6f642ac8395c928da1c58d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49041719e0ae42a390452b8716cc3ca7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:08.433064 11765 raft_consensus.cc:1261] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240502 14:00:08.434350 13686 consensus_queue.cc:1035] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 62, Time since last communication: 0.000s
I20240502 14:00:08.440100 13686 raft_consensus.cc:2932] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Committing config change with OpId 1.64: config changed from index -1 to 64, NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } } }
I20240502 14:00:08.446593 11767 raft_consensus.cc:2932] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index -1 to 64, NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } } }
I20240502 14:00:08.456092 13718 ts_tablet_manager.cc:902] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28: Initiating tablet copy from peer 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 14:00:08.459883 13718 tablet_copy_client.cc:308] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:39411
I20240502 14:00:08.486521 11788 tablet_copy_service.cc:140] P 73ce3d001cdf4d0c8e1487958b7694ea: Received BeginTabletCopySession request for tablet b9a6f7fca259430dac2ca17a8b180b2a from peer 4f6f2d2b1569415cbbbea2466dc28e28 ({username='slave'} at 127.8.86.132:45373)
I20240502 14:00:08.486708 11788 tablet_copy_service.cc:161] P 73ce3d001cdf4d0c8e1487958b7694ea: Beginning new tablet copy session on tablet b9a6f7fca259430dac2ca17a8b180b2a from peer 4f6f2d2b1569415cbbbea2466dc28e28 at {username='slave'} at 127.8.86.132:45373: session id = 4f6f2d2b1569415cbbbea2466dc28e28-b9a6f7fca259430dac2ca17a8b180b2a
I20240502 14:00:08.488312 11788 tablet_copy_source_session.cc:205] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:08.491020 11624 catalog_manager.cc:5484] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: config changed from index -1 to 64, NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "65a078666d30488f9c204d93e6394ef9" committed_config { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } } }
I20240502 14:00:08.494581 13706 tablet_bootstrap.cc:492] T 7cd1b1872364411e86b810420c54034a P 5fbb448689c949d98dec8e37cf5c3478: Bootstrap starting.
I20240502 14:00:08.498878 13718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9a6f7fca259430dac2ca17a8b180b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:08.507303 13718 tablet_copy_client.cc:783] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:08.507975 13718 tablet_copy_client.cc:647] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:08.514134 13718 tablet_copy_client.cc:523] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:08.525321 13706 log.cc:824] T 7cd1b1872364411e86b810420c54034a P 5fbb448689c949d98dec8e37cf5c3478: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:08.530126 13730 ts_tablet_manager.cc:902] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766: Initiating tablet copy from peer 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 14:00:08.530478 12028 consensus_queue.cc:237] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } }
I20240502 14:00:08.531930 13730 tablet_copy_client.cc:308] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:39411
I20240502 14:00:08.536228 11899 raft_consensus.cc:1261] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 40. (index mismatch)
I20240502 14:00:08.538791 11788 tablet_copy_service.cc:140] P 73ce3d001cdf4d0c8e1487958b7694ea: Received BeginTabletCopySession request for tablet c951304cdaa54040a81e51d076fdeaaf from peer a36d5a88b1a34006bd3d040f5a715766 ({username='slave'} at 127.8.86.134:42887)
I20240502 14:00:08.538900 11788 tablet_copy_service.cc:161] P 73ce3d001cdf4d0c8e1487958b7694ea: Beginning new tablet copy session on tablet c951304cdaa54040a81e51d076fdeaaf from peer a36d5a88b1a34006bd3d040f5a715766 at {username='slave'} at 127.8.86.134:42887: session id = a36d5a88b1a34006bd3d040f5a715766-c951304cdaa54040a81e51d076fdeaaf
I20240502 14:00:08.539456 13686 consensus_queue.cc:1035] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20240502 14:00:08.539914 11788 tablet_copy_source_session.cc:205] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:08.543241 13518 raft_consensus.cc:2932] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Committing config change with OpId 1.40: config changed from index -1 to 40, NON_VOTER 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } } }
I20240502 14:00:08.544181 13730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c951304cdaa54040a81e51d076fdeaaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:08.545181 11898 raft_consensus.cc:2932] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Committing config change with OpId 1.40: config changed from index -1 to 40, NON_VOTER 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } } }
I20240502 14:00:08.554303 13730 tablet_copy_client.cc:783] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:08.554627 13730 tablet_copy_client.cc:647] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:08.557014 13730 tablet_copy_client.cc:523] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 14:00:08.579077 11966 consensus_peers.cc:485] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 -> Peer 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132:42709): Couldn't send request to peer 4f6f2d2b1569415cbbbea2466dc28e28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1870fb0c459f49cab36812d16e30ddf5. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:08.585374 13739 ts_tablet_manager.cc:902] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478: Initiating tablet copy from peer cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377)
I20240502 14:00:08.587397 13739 tablet_copy_client.cc:308] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:44377
I20240502 14:00:08.587426 13718 tablet_bootstrap.cc:492] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28: Bootstrap starting.
I20240502 14:00:08.587915 11919 tablet_copy_service.cc:140] P cbf92f56d7144a22a6d482c73c7c7ea9: Received BeginTabletCopySession request for tablet 03409b5f413f499c919f2c6a5a4126f7 from peer 5fbb448689c949d98dec8e37cf5c3478 ({username='slave'} at 127.8.86.133:34065)
I20240502 14:00:08.588013 11919 tablet_copy_service.cc:161] P cbf92f56d7144a22a6d482c73c7c7ea9: Beginning new tablet copy session on tablet 03409b5f413f499c919f2c6a5a4126f7 from peer 5fbb448689c949d98dec8e37cf5c3478 at {username='slave'} at 127.8.86.133:34065: session id = 5fbb448689c949d98dec8e37cf5c3478-03409b5f413f499c919f2c6a5a4126f7
I20240502 14:00:08.588917 11919 tablet_copy_source_session.cc:205] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:08.591127 11624 catalog_manager.cc:5484] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 reported cstate change: config changed from index -1 to 40, NON_VOTER 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) added. New cstate: current_term: 1 leader_uuid: "65a078666d30488f9c204d93e6394ef9" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:08.595301 13738 ts_tablet_manager.cc:902] T c0230720456b49339caaa9d313b6ce64 P 7553d1bab6f642ac8395c928da1c58d7: Initiating tablet copy from peer 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:08.597532 13738 tablet_copy_client.cc:308] T c0230720456b49339caaa9d313b6ce64 P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:45325
I20240502 14:00:08.602154 13739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03409b5f413f499c919f2c6a5a4126f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:08.604631 13739 tablet_copy_client.cc:783] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:08.604879 13739 tablet_copy_client.cc:647] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:08.611521 13730 tablet_bootstrap.cc:492] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766: Bootstrap starting.
I20240502 14:00:08.625600 13739 tablet_copy_client.cc:523] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:08.631525 13706 tablet_bootstrap.cc:492] T 7cd1b1872364411e86b810420c54034a P 5fbb448689c949d98dec8e37cf5c3478: Bootstrap replayed 1/1 log segments. Stats: ops{read=51 overwritten=0 applied=51 ignored=0} inserts{seen=682 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:08.631937 13706 tablet_bootstrap.cc:492] T 7cd1b1872364411e86b810420c54034a P 5fbb448689c949d98dec8e37cf5c3478: Bootstrap complete.
I20240502 14:00:08.632298 13706 ts_tablet_manager.cc:1367] T 7cd1b1872364411e86b810420c54034a P 5fbb448689c949d98dec8e37cf5c3478: Time spent bootstrapping tablet: real 0.138s user 0.062s sys 0.008s
I20240502 14:00:08.635202 13706 raft_consensus.cc:348] T 7cd1b1872364411e86b810420c54034a P 5fbb448689c949d98dec8e37cf5c3478 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } }
I20240502 14:00:08.635669 13706 raft_consensus.cc:729] T 7cd1b1872364411e86b810420c54034a P 5fbb448689c949d98dec8e37cf5c3478 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5fbb448689c949d98dec8e37cf5c3478, State: Initialized, Role: LEARNER
I20240502 14:00:08.636104 13706 consensus_queue.cc:260] T 7cd1b1872364411e86b810420c54034a P 5fbb448689c949d98dec8e37cf5c3478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 51, Last appended: 1.51, Last appended by leader: 51, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } }
I20240502 14:00:08.637195 13706 ts_tablet_manager.cc:1398] T 7cd1b1872364411e86b810420c54034a P 5fbb448689c949d98dec8e37cf5c3478: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 14:00:08.637998 11919 tablet_copy_service.cc:342] P cbf92f56d7144a22a6d482c73c7c7ea9: Request end of tablet copy session 5fbb448689c949d98dec8e37cf5c3478-7cd1b1872364411e86b810420c54034a received from {username='slave'} at 127.8.86.133:34065
I20240502 14:00:08.638101 11919 tablet_copy_service.cc:433] P cbf92f56d7144a22a6d482c73c7c7ea9: ending tablet copy session 5fbb448689c949d98dec8e37cf5c3478-7cd1b1872364411e86b810420c54034a on tablet 7cd1b1872364411e86b810420c54034a with peer 5fbb448689c949d98dec8e37cf5c3478
I20240502 14:00:08.641505 11767 consensus_queue.cc:237] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 1.79, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } }
I20240502 14:00:08.642830 11898 raft_consensus.cc:1261] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240502 14:00:08.644461 13718 log.cc:824] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:08.644749 13506 consensus_queue.cc:1035] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20240502 14:00:08.648778 13506 raft_consensus.cc:2932] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Committing config change with OpId 1.80: config changed from index -1 to 80, NON_VOTER 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) added. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } } }
I20240502 14:00:08.649102 11898 raft_consensus.cc:2932] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index -1 to 80, NON_VOTER 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) added. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } } }
I20240502 14:00:08.650178 13730 log.cc:824] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:08.655292 13760 ts_tablet_manager.cc:902] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28: Initiating tablet copy from peer 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 14:00:08.656160 13760 tablet_copy_client.cc:308] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:39411
I20240502 14:00:08.658802 11788 tablet_copy_service.cc:140] P 73ce3d001cdf4d0c8e1487958b7694ea: Received BeginTabletCopySession request for tablet 11f585c28308464bb46aabdf0ad98a08 from peer 4f6f2d2b1569415cbbbea2466dc28e28 ({username='slave'} at 127.8.86.132:45373)
I20240502 14:00:08.658911 11788 tablet_copy_service.cc:161] P 73ce3d001cdf4d0c8e1487958b7694ea: Beginning new tablet copy session on tablet 11f585c28308464bb46aabdf0ad98a08 from peer 4f6f2d2b1569415cbbbea2466dc28e28 at {username='slave'} at 127.8.86.132:45373: session id = 4f6f2d2b1569415cbbbea2466dc28e28-11f585c28308464bb46aabdf0ad98a08
I20240502 14:00:08.659909 11788 tablet_copy_source_session.cc:205] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:08.661634 13760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f585c28308464bb46aabdf0ad98a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:08.664690 13760 tablet_copy_client.cc:783] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:08.664916 13760 tablet_copy_client.cc:647] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:08.666698 13760 tablet_copy_client.cc:523] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:08.672046 12050 tablet_copy_service.cc:140] P 65a078666d30488f9c204d93e6394ef9: Received BeginTabletCopySession request for tablet c0230720456b49339caaa9d313b6ce64 from peer 7553d1bab6f642ac8395c928da1c58d7 ({username='slave'} at 127.8.86.135:36049)
I20240502 14:00:08.672227 12050 tablet_copy_service.cc:161] P 65a078666d30488f9c204d93e6394ef9: Beginning new tablet copy session on tablet c0230720456b49339caaa9d313b6ce64 from peer 7553d1bab6f642ac8395c928da1c58d7 at {username='slave'} at 127.8.86.135:36049: session id = 7553d1bab6f642ac8395c928da1c58d7-c0230720456b49339caaa9d313b6ce64
I20240502 14:00:08.673604 12050 tablet_copy_source_session.cc:205] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 14:00:08.675997 11703 consensus_peers.cc:485] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea -> Peer 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133:44919): Couldn't send request to peer 5fbb448689c949d98dec8e37cf5c3478. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a50cea2712d48ca812cea7983d16840. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:08.679390 13738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0230720456b49339caaa9d313b6ce64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:08.682332 13739 tablet_bootstrap.cc:492] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478: Bootstrap starting.
I20240502 14:00:08.684247 13760 tablet_bootstrap.cc:492] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28: Bootstrap starting.
I20240502 14:00:08.686435 13738 tablet_copy_client.cc:783] T c0230720456b49339caaa9d313b6ce64 P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:08.686663 13738 tablet_copy_client.cc:647] T c0230720456b49339caaa9d313b6ce64 P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:08.690609 11619 catalog_manager.cc:5484] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: config changed from index -1 to 80, NON_VOTER 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } } }
I20240502 14:00:08.696446 13738 tablet_copy_client.cc:523] T c0230720456b49339caaa9d313b6ce64 P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:08.698525 13739 tablet_bootstrap.cc:492] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:08.698810 13739 tablet_bootstrap.cc:492] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478: Bootstrap complete.
I20240502 14:00:08.699007 13739 ts_tablet_manager.cc:1367] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240502 14:00:08.699616 13739 raft_consensus.cc:348] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } }
I20240502 14:00:08.699764 13739 raft_consensus.cc:729] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5fbb448689c949d98dec8e37cf5c3478, State: Initialized, Role: LEARNER
I20240502 14:00:08.699867 13739 consensus_queue.cc:260] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } }
I20240502 14:00:08.700130 13739 ts_tablet_manager.cc:1398] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:08.701197 11919 tablet_copy_service.cc:342] P cbf92f56d7144a22a6d482c73c7c7ea9: Request end of tablet copy session 5fbb448689c949d98dec8e37cf5c3478-03409b5f413f499c919f2c6a5a4126f7 received from {username='slave'} at 127.8.86.133:34065
I20240502 14:00:08.701292 11919 tablet_copy_service.cc:433] P cbf92f56d7144a22a6d482c73c7c7ea9: ending tablet copy session 5fbb448689c949d98dec8e37cf5c3478-03409b5f413f499c919f2c6a5a4126f7 on tablet 03409b5f413f499c919f2c6a5a4126f7 with peer 5fbb448689c949d98dec8e37cf5c3478
I20240502 14:00:08.707445 11898 consensus_queue.cc:237] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 1.45, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } }
I20240502 14:00:08.715050 11767 raft_consensus.cc:1261] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 45. Preceding OpId from leader: term: 1 index: 46. (index mismatch)
I20240502 14:00:08.720626 13564 consensus_queue.cc:1035] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20240502 14:00:08.735751 13564 raft_consensus.cc:2932] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } } }
I20240502 14:00:08.738685 11767 raft_consensus.cc:2932] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } } }
W20240502 14:00:08.748142 11835 consensus_peers.cc:485] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 -> Peer 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135:34019): Couldn't send request to peer 7553d1bab6f642ac8395c928da1c58d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 236933482b6b41cf9e84a2eeb3b57638. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:08.778179 13738 tablet_bootstrap.cc:492] T c0230720456b49339caaa9d313b6ce64 P 7553d1bab6f642ac8395c928da1c58d7: Bootstrap starting.
I20240502 14:00:08.782353 13760 tablet_bootstrap.cc:492] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28: Bootstrap replayed 1/1 log segments. Stats: ops{read=67 overwritten=0 applied=66 ignored=0} inserts{seen=738 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:00:08.782795 13760 tablet_bootstrap.cc:492] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28: Bootstrap complete.
I20240502 14:00:08.783275 11624 catalog_manager.cc:5484] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: config changed from index -1 to 46, NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:08.784534 13760 ts_tablet_manager.cc:1367] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28: Time spent bootstrapping tablet: real 0.100s user 0.071s sys 0.000s
I20240502 14:00:08.795620 13760 raft_consensus.cc:348] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } }
I20240502 14:00:08.797813 13760 raft_consensus.cc:729] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f6f2d2b1569415cbbbea2466dc28e28, State: Initialized, Role: LEARNER
I20240502 14:00:08.798204 13760 consensus_queue.cc:260] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 66, Last appended: 1.67, Last appended by leader: 67, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } }
I20240502 14:00:08.799172 13760 ts_tablet_manager.cc:1398] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28: Time spent starting tablet: real 0.015s user 0.001s sys 0.004s
I20240502 14:00:08.795046 13738 log.cc:824] T c0230720456b49339caaa9d313b6ce64 P 7553d1bab6f642ac8395c928da1c58d7: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:08.802644 11788 tablet_copy_service.cc:342] P 73ce3d001cdf4d0c8e1487958b7694ea: Request end of tablet copy session 4f6f2d2b1569415cbbbea2466dc28e28-11f585c28308464bb46aabdf0ad98a08 received from {username='slave'} at 127.8.86.132:45373
I20240502 14:00:08.802743 11788 tablet_copy_service.cc:433] P 73ce3d001cdf4d0c8e1487958b7694ea: ending tablet copy session 4f6f2d2b1569415cbbbea2466dc28e28-11f585c28308464bb46aabdf0ad98a08 on tablet 11f585c28308464bb46aabdf0ad98a08 with peer 4f6f2d2b1569415cbbbea2466dc28e28
I20240502 14:00:08.818473 13730 tablet_bootstrap.cc:492] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766: Bootstrap replayed 1/1 log segments. Stats: ops{read=54 overwritten=0 applied=54 ignored=0} inserts{seen=739 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:08.818883 13730 tablet_bootstrap.cc:492] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766: Bootstrap complete.
I20240502 14:00:08.819236 13730 ts_tablet_manager.cc:1367] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766: Time spent bootstrapping tablet: real 0.208s user 0.077s sys 0.000s
I20240502 14:00:08.839118 13718 tablet_bootstrap.cc:492] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28: Bootstrap replayed 1/1 log segments. Stats: ops{read=68 overwritten=0 applied=68 ignored=0} inserts{seen=740 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:08.839468 13718 tablet_bootstrap.cc:492] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28: Bootstrap complete.
I20240502 14:00:08.839660 13718 ts_tablet_manager.cc:1367] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28: Time spent bootstrapping tablet: real 0.252s user 0.082s sys 0.004s
I20240502 14:00:08.840224 13718 raft_consensus.cc:348] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } }
I20240502 14:00:08.840351 13718 raft_consensus.cc:729] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4f6f2d2b1569415cbbbea2466dc28e28, State: Initialized, Role: LEARNER
I20240502 14:00:08.840456 13718 consensus_queue.cc:260] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 68, Last appended: 2.68, Last appended by leader: 68, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } }
I20240502 14:00:08.840751 13718 ts_tablet_manager.cc:1398] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:08.844329 13730 raft_consensus.cc:348] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } }
I20240502 14:00:08.844666 13730 raft_consensus.cc:729] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a36d5a88b1a34006bd3d040f5a715766, State: Initialized, Role: LEARNER
I20240502 14:00:08.845072 13730 consensus_queue.cc:260] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 54, Last appended: 1.54, Last appended by leader: 54, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } }
I20240502 14:00:08.845322 11788 tablet_copy_service.cc:342] P 73ce3d001cdf4d0c8e1487958b7694ea: Request end of tablet copy session 4f6f2d2b1569415cbbbea2466dc28e28-b9a6f7fca259430dac2ca17a8b180b2a received from {username='slave'} at 127.8.86.132:45373
I20240502 14:00:08.845407 11788 tablet_copy_service.cc:433] P 73ce3d001cdf4d0c8e1487958b7694ea: ending tablet copy session 4f6f2d2b1569415cbbbea2466dc28e28-b9a6f7fca259430dac2ca17a8b180b2a on tablet b9a6f7fca259430dac2ca17a8b180b2a with peer 4f6f2d2b1569415cbbbea2466dc28e28
I20240502 14:00:08.846472 13730 ts_tablet_manager.cc:1398] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20240502 14:00:08.847362 11788 tablet_copy_service.cc:342] P 73ce3d001cdf4d0c8e1487958b7694ea: Request end of tablet copy session a36d5a88b1a34006bd3d040f5a715766-c951304cdaa54040a81e51d076fdeaaf received from {username='slave'} at 127.8.86.134:42887
I20240502 14:00:08.847434 11788 tablet_copy_service.cc:433] P 73ce3d001cdf4d0c8e1487958b7694ea: ending tablet copy session a36d5a88b1a34006bd3d040f5a715766-c951304cdaa54040a81e51d076fdeaaf on tablet c951304cdaa54040a81e51d076fdeaaf with peer a36d5a88b1a34006bd3d040f5a715766
I20240502 14:00:08.866132 11767 consensus_queue.cc:237] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 1.55, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } }
I20240502 14:00:08.868285 12028 raft_consensus.cc:1261] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240502 14:00:08.868930 13699 consensus_queue.cc:1035] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20240502 14:00:08.870766 13699 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Committing config change with OpId 1.56: config changed from index -1 to 56, NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } } }
I20240502 14:00:08.872187 12028 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index -1 to 56, NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } } }
W20240502 14:00:08.873652 11704 consensus_peers.cc:485] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea -> Peer 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135:34019): Couldn't send request to peer 7553d1bab6f642ac8395c928da1c58d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9bc1c84cb144965a21bdc40f099416d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:08.886838 11624 catalog_manager.cc:5484] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9 reported cstate change: config changed from index -1 to 56, NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } } }
I20240502 14:00:08.891794 13040 raft_consensus.cc:1203] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.39->[2.40-2.77] Dedup: 2.68->[2.69-2.77]
I20240502 14:00:08.899000 13172 raft_consensus.cc:1203] T 7cd1b1872364411e86b810420c54034a P 5fbb448689c949d98dec8e37cf5c3478 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.36->[1.37-1.64] Dedup: 1.51->[1.52-1.64]
I20240502 14:00:08.924162 13730 ts_tablet_manager.cc:902] T 259e249430fa41a2bc2528b65d022b67 P a36d5a88b1a34006bd3d040f5a715766: Initiating tablet copy from peer 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 14:00:08.924499 13785 ts_tablet_manager.cc:902] T 49041719e0ae42a390452b8716cc3ca7 P 7553d1bab6f642ac8395c928da1c58d7: Initiating tablet copy from peer 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:08.924671 13730 tablet_copy_client.cc:308] T 259e249430fa41a2bc2528b65d022b67 P a36d5a88b1a34006bd3d040f5a715766: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:39411
I20240502 14:00:08.925201 13785 tablet_copy_client.cc:308] T 49041719e0ae42a390452b8716cc3ca7 P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:45325
I20240502 14:00:08.925235 11788 tablet_copy_service.cc:140] P 73ce3d001cdf4d0c8e1487958b7694ea: Received BeginTabletCopySession request for tablet 259e249430fa41a2bc2528b65d022b67 from peer a36d5a88b1a34006bd3d040f5a715766 ({username='slave'} at 127.8.86.134:42887)
I20240502 14:00:08.925344 11788 tablet_copy_service.cc:161] P 73ce3d001cdf4d0c8e1487958b7694ea: Beginning new tablet copy session on tablet 259e249430fa41a2bc2528b65d022b67 from peer a36d5a88b1a34006bd3d040f5a715766 at {username='slave'} at 127.8.86.134:42887: session id = a36d5a88b1a34006bd3d040f5a715766-259e249430fa41a2bc2528b65d022b67
I20240502 14:00:08.926206 12050 tablet_copy_service.cc:140] P 65a078666d30488f9c204d93e6394ef9: Received BeginTabletCopySession request for tablet 49041719e0ae42a390452b8716cc3ca7 from peer 7553d1bab6f642ac8395c928da1c58d7 ({username='slave'} at 127.8.86.135:36049)
I20240502 14:00:08.926321 12050 tablet_copy_service.cc:161] P 65a078666d30488f9c204d93e6394ef9: Beginning new tablet copy session on tablet 49041719e0ae42a390452b8716cc3ca7 from peer 7553d1bab6f642ac8395c928da1c58d7 at {username='slave'} at 127.8.86.135:36049: session id = 7553d1bab6f642ac8395c928da1c58d7-49041719e0ae42a390452b8716cc3ca7
I20240502 14:00:08.926371 11788 tablet_copy_source_session.cc:205] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:08.927246 12050 tablet_copy_source_session.cc:205] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:08.928386 13718 ts_tablet_manager.cc:902] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28: Initiating tablet copy from peer 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:08.928912 13718 tablet_copy_client.cc:308] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:45325
I20240502 14:00:08.930114 13730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 259e249430fa41a2bc2528b65d022b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:08.932475 13730 tablet_copy_client.cc:783] T 259e249430fa41a2bc2528b65d022b67 P a36d5a88b1a34006bd3d040f5a715766: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:08.932627 13730 tablet_copy_client.cc:647] T 259e249430fa41a2bc2528b65d022b67 P a36d5a88b1a34006bd3d040f5a715766: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:08.934444 12050 tablet_copy_service.cc:140] P 65a078666d30488f9c204d93e6394ef9: Received BeginTabletCopySession request for tablet 1870fb0c459f49cab36812d16e30ddf5 from peer 4f6f2d2b1569415cbbbea2466dc28e28 ({username='slave'} at 127.8.86.132:45333)
I20240502 14:00:08.934547 12050 tablet_copy_service.cc:161] P 65a078666d30488f9c204d93e6394ef9: Beginning new tablet copy session on tablet 1870fb0c459f49cab36812d16e30ddf5 from peer 4f6f2d2b1569415cbbbea2466dc28e28 at {username='slave'} at 127.8.86.132:45333: session id = 4f6f2d2b1569415cbbbea2466dc28e28-1870fb0c459f49cab36812d16e30ddf5
I20240502 14:00:08.935392 12050 tablet_copy_source_session.cc:205] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:08.935626 13785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49041719e0ae42a390452b8716cc3ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:08.936266 13718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1870fb0c459f49cab36812d16e30ddf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:08.936347 12861 raft_consensus.cc:1050] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea: attempting to promote NON_VOTER 4f6f2d2b1569415cbbbea2466dc28e28 to VOTER
I20240502 14:00:08.936761 12861 consensus_queue.cc:237] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 2.80, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } }
I20240502 14:00:08.937937 13785 tablet_copy_client.cc:783] T 49041719e0ae42a390452b8716cc3ca7 P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:08.938100 13785 tablet_copy_client.cc:647] T 49041719e0ae42a390452b8716cc3ca7 P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:08.940156 13039 raft_consensus.cc:1261] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28 [term 2 LEARNER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 80. Preceding OpId from leader: term: 2 index: 81. (index mismatch)
I20240502 14:00:08.940830 13559 consensus_queue.cc:1035] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240502 14:00:08.941344 13718 tablet_copy_client.cc:783] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:08.941502 13718 tablet_copy_client.cc:647] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:08.941987 11899 raft_consensus.cc:1261] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 80. Preceding OpId from leader: term: 2 index: 81. (index mismatch)
I20240502 14:00:08.942137 13785 tablet_copy_client.cc:523] T 49041719e0ae42a390452b8716cc3ca7 P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:08.942451 13566 consensus_queue.cc:1035] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240502 14:00:08.945497 13559 raft_consensus.cc:2932] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 LEADER]: Committing config change with OpId 2.81: config changed from index 40 to 81, 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) changed from NON_VOTER to VOTER. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:08.946046 13718 tablet_copy_client.cc:523] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:08.950140 13039 raft_consensus.cc:2932] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28 [term 2 FOLLOWER]: Committing config change with OpId 2.81: config changed from index 40 to 81, 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) changed from NON_VOTER to VOTER. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:08.952791 13730 tablet_copy_client.cc:523] T 259e249430fa41a2bc2528b65d022b67 P a36d5a88b1a34006bd3d040f5a715766: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:08.952674 11899 raft_consensus.cc:2932] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Committing config change with OpId 2.81: config changed from index 40 to 81, 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) changed from NON_VOTER to VOTER. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:08.963528 13738 tablet_bootstrap.cc:492] T c0230720456b49339caaa9d313b6ce64 P 7553d1bab6f642ac8395c928da1c58d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=54 overwritten=0 applied=54 ignored=0} inserts{seen=739 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:08.964078 13738 tablet_bootstrap.cc:492] T c0230720456b49339caaa9d313b6ce64 P 7553d1bab6f642ac8395c928da1c58d7: Bootstrap complete.
I20240502 14:00:08.964499 13738 ts_tablet_manager.cc:1367] T c0230720456b49339caaa9d313b6ce64 P 7553d1bab6f642ac8395c928da1c58d7: Time spent bootstrapping tablet: real 0.187s user 0.067s sys 0.008s
I20240502 14:00:08.967936 13738 raft_consensus.cc:348] T c0230720456b49339caaa9d313b6ce64 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } }
I20240502 14:00:08.968297 13738 raft_consensus.cc:729] T c0230720456b49339caaa9d313b6ce64 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7553d1bab6f642ac8395c928da1c58d7, State: Initialized, Role: LEARNER
I20240502 14:00:08.968735 13738 consensus_queue.cc:260] T c0230720456b49339caaa9d313b6ce64 P 7553d1bab6f642ac8395c928da1c58d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 54, Last appended: 1.54, Last appended by leader: 54, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } }
I20240502 14:00:08.969733 13738 ts_tablet_manager.cc:1398] T c0230720456b49339caaa9d313b6ce64 P 7553d1bab6f642ac8395c928da1c58d7: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240502 14:00:08.971588 12050 tablet_copy_service.cc:342] P 65a078666d30488f9c204d93e6394ef9: Request end of tablet copy session 7553d1bab6f642ac8395c928da1c58d7-c0230720456b49339caaa9d313b6ce64 received from {username='slave'} at 127.8.86.135:36049
I20240502 14:00:08.971685 12050 tablet_copy_service.cc:433] P 65a078666d30488f9c204d93e6394ef9: ending tablet copy session 7553d1bab6f642ac8395c928da1c58d7-c0230720456b49339caaa9d313b6ce64 on tablet c0230720456b49339caaa9d313b6ce64 with peer 7553d1bab6f642ac8395c928da1c58d7
I20240502 14:00:08.978621 13785 tablet_bootstrap.cc:492] T 49041719e0ae42a390452b8716cc3ca7 P 7553d1bab6f642ac8395c928da1c58d7: Bootstrap starting.
I20240502 14:00:08.978632 13730 tablet_bootstrap.cc:492] T 259e249430fa41a2bc2528b65d022b67 P a36d5a88b1a34006bd3d040f5a715766: Bootstrap starting.
I20240502 14:00:08.982782 13718 tablet_bootstrap.cc:492] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28: Bootstrap starting.
I20240502 14:00:08.985271 11619 catalog_manager.cc:5484] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28 reported cstate change: config changed from index 40 to 81, 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:08.994251 11767 consensus_queue.cc:237] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 2.82, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } }
I20240502 14:00:09.006898 13039 raft_consensus.cc:1261] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28 [term 2 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 82. Preceding OpId from leader: term: 2 index: 83. (index mismatch)
I20240502 14:00:09.007550 13699 consensus_queue.cc:1035] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20240502 14:00:09.011961 13559 raft_consensus.cc:2932] T b9a6f7fca259430dac2ca17a8b180b2a P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 LEADER]: Committing config change with OpId 2.83: config changed from index 81 to 83, VOTER cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) evicted. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:09.012450 13039 raft_consensus.cc:2932] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28 [term 2 FOLLOWER]: Committing config change with OpId 2.83: config changed from index 81 to 83, VOTER cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) evicted. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:09.017625 11619 catalog_manager.cc:5484] T b9a6f7fca259430dac2ca17a8b180b2a P 4f6f2d2b1569415cbbbea2466dc28e28 reported cstate change: config changed from index 81 to 83, VOTER cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) evicted. New cstate: current_term: 2 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:09.020927 11877 tablet_service.cc:1514] Processing DeleteTablet for tablet b9a6f7fca259430dac2ca17a8b180b2a with delete_type TABLET_DATA_TOMBSTONED (TS cbf92f56d7144a22a6d482c73c7c7ea9 not found in new config with opid_index 83) from {username='slave'} at 127.0.0.1:37684
I20240502 14:00:09.030683 11612 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b9a6f7fca259430dac2ca17a8b180b2a with cas_config_opid_index 81: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:09.034121 13661 raft_consensus.cc:1050] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9: attempting to promote NON_VOTER 5fbb448689c949d98dec8e37cf5c3478 to VOTER
I20240502 14:00:09.046353 13661 consensus_queue.cc:237] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 1.67, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } }
I20240502 14:00:09.050251 13172 raft_consensus.cc:1261] T 7cd1b1872364411e86b810420c54034a P 5fbb448689c949d98dec8e37cf5c3478 [term 1 LEARNER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch)
I20240502 14:00:09.052762 13572 consensus_queue.cc:1035] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 66, Time since last communication: 0.000s
I20240502 14:00:09.055843 13661 raft_consensus.cc:2932] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Committing config change with OpId 1.68: config changed from index 37 to 68, 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
I20240502 14:00:09.059689 11620 catalog_manager.cc:5484] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: config changed from index 37 to 68, 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:09.059844 13172 raft_consensus.cc:2932] T 7cd1b1872364411e86b810420c54034a P 5fbb448689c949d98dec8e37cf5c3478 [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index 37 to 68, 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
I20240502 14:00:09.067848 11899 consensus_queue.cc:237] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } }
I20240502 14:00:09.070091 13172 raft_consensus.cc:1261] T 7cd1b1872364411e86b810420c54034a P 5fbb448689c949d98dec8e37cf5c3478 [term 1 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240502 14:00:09.070698 13783 consensus_queue.cc:1035] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 68, Time since last communication: 0.000s
I20240502 14:00:09.072201 13039 raft_consensus.cc:1203] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.49->[1.50-1.75] Dedup: 1.67->[1.68-1.75]
I20240502 14:00:09.073421 13661 raft_consensus.cc:2932] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index 68 to 70, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
I20240502 14:00:09.075340 11613 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7cd1b1872364411e86b810420c54034a with cas_config_opid_index 68: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:09.078172 11619 catalog_manager.cc:5484] T 7cd1b1872364411e86b810420c54034a P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: config changed from index 68 to 70, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New cstate: current_term: 1 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:09.079516 13172 raft_consensus.cc:2932] T 7cd1b1872364411e86b810420c54034a P 5fbb448689c949d98dec8e37cf5c3478 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 68 to 70, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
I20240502 14:00:09.081511 12026 raft_consensus.cc:1261] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch)
I20240502 14:00:09.088277 13569 raft_consensus.cc:1050] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea: attempting to promote NON_VOTER 4f6f2d2b1569415cbbbea2466dc28e28 to VOTER
I20240502 14:00:09.088667 13569 consensus_queue.cc:237] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.76, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } }
I20240502 14:00:09.089052 12008 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cd1b1872364411e86b810420c54034a with delete_type TABLET_DATA_TOMBSTONED (TS 65a078666d30488f9c204d93e6394ef9 not found in new config with opid_index 70) from {username='slave'} at 127.0.0.1:38592
I20240502 14:00:09.090167 13040 raft_consensus.cc:1261] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 LEARNER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
I20240502 14:00:09.092793 13303 raft_consensus.cc:1203] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.34->[1.35-1.65] Dedup: 1.54->[1.55-1.65]
I20240502 14:00:09.094962 13506 consensus_queue.cc:1035] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240502 14:00:09.097589 12026 raft_consensus.cc:1261] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
I20240502 14:00:09.098001 13699 consensus_queue.cc:1035] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240502 14:00:09.101027 12861 raft_consensus.cc:2932] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Committing config change with OpId 1.77: config changed from index 50 to 77, 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) changed from NON_VOTER to VOTER. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:09.101465 13785 tablet_bootstrap.cc:492] T 49041719e0ae42a390452b8716cc3ca7 P 7553d1bab6f642ac8395c928da1c58d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=75 overwritten=0 applied=75 ignored=0} inserts{seen=747 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:09.101822 13785 tablet_bootstrap.cc:492] T 49041719e0ae42a390452b8716cc3ca7 P 7553d1bab6f642ac8395c928da1c58d7: Bootstrap complete.
I20240502 14:00:09.102026 13785 ts_tablet_manager.cc:1367] T 49041719e0ae42a390452b8716cc3ca7 P 7553d1bab6f642ac8395c928da1c58d7: Time spent bootstrapping tablet: real 0.123s user 0.075s sys 0.003s
I20240502 14:00:09.103188 12027 raft_consensus.cc:2932] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index 50 to 77, 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) changed from NON_VOTER to VOTER. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:09.106946 13785 raft_consensus.cc:348] T 49041719e0ae42a390452b8716cc3ca7 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } }
I20240502 14:00:09.107084 13785 raft_consensus.cc:729] T 49041719e0ae42a390452b8716cc3ca7 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7553d1bab6f642ac8395c928da1c58d7, State: Initialized, Role: LEARNER
I20240502 14:00:09.107200 13785 consensus_queue.cc:260] T 49041719e0ae42a390452b8716cc3ca7 P 7553d1bab6f642ac8395c928da1c58d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 75, Last appended: 1.75, Last appended by leader: 75, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } }
I20240502 14:00:09.105826 11620 catalog_manager.cc:5484] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: config changed from index 50 to 77, 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:09.108187 13039 raft_consensus.cc:2932] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index 50 to 77, 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) changed from NON_VOTER to VOTER. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:09.113102 13785 ts_tablet_manager.cc:1398] T 49041719e0ae42a390452b8716cc3ca7 P 7553d1bab6f642ac8395c928da1c58d7: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240502 14:00:09.116672 12050 tablet_copy_service.cc:342] P 65a078666d30488f9c204d93e6394ef9: Request end of tablet copy session 7553d1bab6f642ac8395c928da1c58d7-49041719e0ae42a390452b8716cc3ca7 received from {username='slave'} at 127.8.86.135:36049
I20240502 14:00:09.116777 12050 tablet_copy_service.cc:433] P 65a078666d30488f9c204d93e6394ef9: ending tablet copy session 7553d1bab6f642ac8395c928da1c58d7-49041719e0ae42a390452b8716cc3ca7 on tablet 49041719e0ae42a390452b8716cc3ca7 with peer 7553d1bab6f642ac8395c928da1c58d7
I20240502 14:00:09.124059 11765 consensus_queue.cc:237] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } }
I20240502 14:00:09.127562 13836 tablet_replica.cc:330] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9: stopping tablet replica
I20240502 14:00:09.127987 13836 raft_consensus.cc:2229] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:09.128429 13836 pending_rounds.cc:62] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9: Trying to abort 1 pending ops.
I20240502 14:00:09.128998 13038 raft_consensus.cc:1261] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 77. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240502 14:00:09.130331 13836 pending_rounds.cc:68] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9: Aborting op as it isn't in flight: id { term: 1 index: 67 } timestamp: 7023240843307089920 op_type: WRITE_OP write_request { tablet_id: "7cd1b1872364411e86b810420c54034a" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\265\"\217\tp\222,B\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""ebbbc75dea3bc3d9" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023240843246243840 authz_token { token_data: "\010\222\273\316\261\006\"3\n\005slave\022*\n d77f8a4d2b7c4f42901744c7061e0d56\020\001\030\001 \001(\001" signature: "<redacted>""Z\222\311v`\252\326l$\372\032\007|E\321\177\304>\027xD\277\374\210\247\357\367\312\244\256x\016\217lO\305~\202yg\034\236\337\342W(w\261\274\201f6\366\024\265\251.]\3457\273\037~\303" signing_key_seq_num: 0 } } request_id { client_id: "e03da13323b74ea893b234362cc3b589" seq_no: 194 first_incomplete_seq_no: 194 attempt_no: 0 }
I20240502 14:00:09.132375 13569 consensus_queue.cc:1035] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 77, Time since last communication: 0.000s
I20240502 14:00:09.133592 13836 raft_consensus.cc:2258] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:09.134805 12861 raft_consensus.cc:1050] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea: attempting to promote NON_VOTER a36d5a88b1a34006bd3d040f5a715766 to VOTER
I20240502 14:00:09.135191 12861 consensus_queue.cc:237] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } }
I20240502 14:00:09.136314 12026 raft_consensus.cc:1261] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240502 14:00:09.136344 13172 raft_consensus.cc:1203] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:00:09.136988 13303 raft_consensus.cc:1261] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766 [term 1 LEARNER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240502 14:00:09.137243 13039 raft_consensus.cc:1203] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.77->[1.78-1.78] Dedup: 1.78->[]
I20240502 14:00:09.137774 13700 consensus_queue.cc:1035] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240502 14:00:09.138924 13700 consensus_queue.cc:1035] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240502 14:00:09.139498 13559 raft_consensus.cc:2932] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index 77 to 79, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:09.140105 13506 raft_consensus.cc:2932] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index 35 to 66, a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:09.141840 13303 raft_consensus.cc:2932] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 35 to 66, a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:09.142210 11612 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 11f585c28308464bb46aabdf0ad98a08 with cas_config_opid_index 77: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:09.142755 13039 raft_consensus.cc:2932] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index 77 to 79, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:09.143265 11617 catalog_manager.cc:5484] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: config changed from index 77 to 79, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:09.140983 13836 ts_tablet_manager.cc:1875] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:09.145228 11620 catalog_manager.cc:5484] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766 reported cstate change: config changed from index 35 to 66, a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:09.146886 12008 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f585c28308464bb46aabdf0ad98a08 with delete_type TABLET_DATA_TOMBSTONED (TS 65a078666d30488f9c204d93e6394ef9 not found in new config with opid_index 79) from {username='slave'} at 127.0.0.1:38592
I20240502 14:00:09.149883 11765 consensus_queue.cc:237] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 1.67, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } }
I20240502 14:00:09.151523 13303 raft_consensus.cc:1261] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20240502 14:00:09.152530 13566 consensus_queue.cc:1035] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20240502 14:00:09.157512 13700 raft_consensus.cc:2932] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Committing config change with OpId 1.68: config changed from index 66 to 68, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:09.161057 11612 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c951304cdaa54040a81e51d076fdeaaf with cas_config_opid_index 66: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:09.161376 13836 ts_tablet_manager.cc:1888] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.67
I20240502 14:00:09.161671 11619 catalog_manager.cc:5484] T c951304cdaa54040a81e51d076fdeaaf P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: config changed from index 66 to 68, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:09.163812 13434 raft_consensus.cc:1203] T c0230720456b49339caaa9d313b6ce64 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.31->[1.32-1.68] Dedup: 1.54->[1.55-1.68]
I20240502 14:00:09.161469 13836 log.cc:1196] T 7cd1b1872364411e86b810420c54034a P 65a078666d30488f9c204d93e6394ef9: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/wals/7cd1b1872364411e86b810420c54034a
I20240502 14:00:09.167174 13836 tablet_replica.cc:330] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9: stopping tablet replica
I20240502 14:00:09.167222 11612 catalog_manager.cc:4830] TS 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325): tablet 7cd1b1872364411e86b810420c54034a (table rebalance_test_table_2 [id=d77f8a4d2b7c4f42901744c7061e0d56]) successfully deleted
I20240502 14:00:09.170024 13303 raft_consensus.cc:2932] T c951304cdaa54040a81e51d076fdeaaf P a36d5a88b1a34006bd3d040f5a715766 [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index 66 to 68, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:09.169569 12008 tablet_service.cc:1514] Processing DeleteTablet for tablet c951304cdaa54040a81e51d076fdeaaf with delete_type TABLET_DATA_TOMBSTONED (TS 65a078666d30488f9c204d93e6394ef9 not found in new config with opid_index 68) from {username='slave'} at 127.0.0.1:38592
I20240502 14:00:09.171329 13836 raft_consensus.cc:2229] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:09.171485 13836 pending_rounds.cc:62] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9: Trying to abort 1 pending ops.
I20240502 14:00:09.171564 13836 pending_rounds.cc:68] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9: Aborting op as it isn't in flight: id { term: 1 index: 78 } timestamp: 7023240843767545856 op_type: WRITE_OP write_request { tablet_id: "11f585c28308464bb46aabdf0ad98a08" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\301\204dX\0135\253\221\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""7adb332d99fa1cbd" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023240843740381184 authz_token { token_data: "\010\222\273\316\261\006\"3\n\005slave\022*\n d77f8a4d2b7c4f42901744c7061e0d56\020\001\030\001 \001(\001" signature: "<redacted>""Z\222\311v`\252\326l$\372\032\007|E\321\177\304>\027xD\277\374\210\247\357\367\312\244\256x\016\217lO\305~\202yg\034\236\337\342W(w\261\274\201f6\366\024\265\251.]\3457\273\037~\303" signing_key_seq_num: 0 } } request_id { client_id: "e03da13323b74ea893b234362cc3b589" seq_no: 202 first_incomplete_seq_no: 202 attempt_no: 0 }
I20240502 14:00:09.171985 13836 raft_consensus.cc:2258] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:09.173700 13836 ts_tablet_manager.cc:1875] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:09.180356 13739 ts_tablet_manager.cc:902] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478: Initiating tablet copy from peer 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 14:00:09.180852 13739 tablet_copy_client.cc:308] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:39411
I20240502 14:00:09.184242 13836 ts_tablet_manager.cc:1888] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.78
I20240502 14:00:09.192466 13836 log.cc:1196] T 11f585c28308464bb46aabdf0ad98a08 P 65a078666d30488f9c204d93e6394ef9: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/wals/11f585c28308464bb46aabdf0ad98a08
I20240502 14:00:09.195747 13836 tablet_replica.cc:330] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9: stopping tablet replica
I20240502 14:00:09.196274 11612 catalog_manager.cc:4830] TS 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325): tablet 11f585c28308464bb46aabdf0ad98a08 (table rebalance_test_table_2 [id=d77f8a4d2b7c4f42901744c7061e0d56]) successfully deleted
I20240502 14:00:09.196537 13836 raft_consensus.cc:2229] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:09.197077 13836 pending_rounds.cc:62] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9: Trying to abort 2 pending ops.
I20240502 14:00:09.197160 13836 pending_rounds.cc:68] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9: Aborting op as it isn't in flight: id { term: 1 index: 67 } timestamp: 7023240843831091200 op_type: WRITE_OP write_request { tablet_id: "c951304cdaa54040a81e51d076fdeaaf" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000~\220l\027\370XY\322\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""d734f7f1ad0437e3" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023240843555971072 authz_token { token_data: "\010\222\273\316\261\006\"3\n\005slave\022*\n 610fb42d93954513952217d0731890d9\020\001\030\001 \001(\001" signature: "<redacted>""Y\006T32\331ur>\246P\375VV\324?\327\210\332\336G\247\013<\t\336tN,\276\3252\354-V\264?\355:\031\202E\315\375\206\226\n\325\223.\003\220{\\\321,M\031|\311Nf\351\023" signing_key_seq_num: 0 } } request_id { client_id: "a5be2fb20e0147bb9374c03c081cb466" seq_no: 226 first_incomplete_seq_no: 226 attempt_no: 0 }
I20240502 14:00:09.200688 13836 pending_rounds.cc:68] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9: Aborting op as it isn't in flight: id { term: 1 index: 66 } timestamp: 7023240843816771584 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "c951304cdaa54040a81e51d076fdeaaf" old_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } } } new_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } } }
I20240502 14:00:09.201212 13836 raft_consensus.cc:2894] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Aborting config change with OpId 1.66: Aborted: Op aborted
I20240502 14:00:09.201340 13836 raft_consensus.cc:2866] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:00:09.201444 13836 raft_consensus.cc:2258] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:09.204070 11788 tablet_copy_service.cc:140] P 73ce3d001cdf4d0c8e1487958b7694ea: Received BeginTabletCopySession request for tablet 3a50cea2712d48ca812cea7983d16840 from peer 5fbb448689c949d98dec8e37cf5c3478 ({username='slave'} at 127.8.86.133:43393)
I20240502 14:00:09.204170 11788 tablet_copy_service.cc:161] P 73ce3d001cdf4d0c8e1487958b7694ea: Beginning new tablet copy session on tablet 3a50cea2712d48ca812cea7983d16840 from peer 5fbb448689c949d98dec8e37cf5c3478 at {username='slave'} at 127.8.86.133:43393: session id = 5fbb448689c949d98dec8e37cf5c3478-3a50cea2712d48ca812cea7983d16840
I20240502 14:00:09.205133 11788 tablet_copy_source_session.cc:205] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:09.206477 13739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a50cea2712d48ca812cea7983d16840. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:09.208755 13739 tablet_copy_client.cc:783] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:09.208904 13739 tablet_copy_client.cc:647] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:09.213546 13739 tablet_copy_client.cc:523] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:09.213707 13836 ts_tablet_manager.cc:1875] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:09.214969 13739 tablet_bootstrap.cc:492] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478: Bootstrap starting.
I20240502 14:00:09.220064 13785 ts_tablet_manager.cc:902] T 236933482b6b41cf9e84a2eeb3b57638 P 7553d1bab6f642ac8395c928da1c58d7: Initiating tablet copy from peer cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377)
I20240502 14:00:09.220618 13785 tablet_copy_client.cc:308] T 236933482b6b41cf9e84a2eeb3b57638 P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:44377
I20240502 14:00:09.224516 13823 tablet_replica.cc:330] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9: stopping tablet replica
I20240502 14:00:09.224854 13823 raft_consensus.cc:2229] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:09.225116 13823 raft_consensus.cc:2258] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:09.228744 13823 ts_tablet_manager.cc:1875] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:09.231712 13617 raft_consensus.cc:1050] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9: attempting to promote NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 to VOTER
I20240502 14:00:09.232116 13617 consensus_queue.cc:237] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 1.72, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } }
I20240502 14:00:09.232817 13836 ts_tablet_manager.cc:1888] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.67
I20240502 14:00:09.232899 13836 log.cc:1196] T c951304cdaa54040a81e51d076fdeaaf P 65a078666d30488f9c204d93e6394ef9: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/wals/c951304cdaa54040a81e51d076fdeaaf
I20240502 14:00:09.234385 11767 raft_consensus.cc:1261] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 1 index: 73. (index mismatch)
I20240502 14:00:09.234843 13433 raft_consensus.cc:1261] T c0230720456b49339caaa9d313b6ce64 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 LEARNER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 1 index: 73. (index mismatch)
I20240502 14:00:09.236208 13518 consensus_queue.cc:1035] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20240502 14:00:09.236600 11612 catalog_manager.cc:4830] TS 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325): tablet c951304cdaa54040a81e51d076fdeaaf (table rebalance_test_table_0 [id=610fb42d93954513952217d0731890d9]) successfully deleted
I20240502 14:00:09.237888 13518 consensus_queue.cc:1035] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20240502 14:00:09.241786 13518 raft_consensus.cc:2932] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Committing config change with OpId 1.73: config changed from index 32 to 73, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.244297 11767 raft_consensus.cc:2932] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Committing config change with OpId 1.73: config changed from index 32 to 73, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.244472 11619 catalog_manager.cc:5484] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 reported cstate change: config changed from index 32 to 73, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "65a078666d30488f9c204d93e6394ef9" committed_config { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:09.244307 13433 raft_consensus.cc:2932] T c0230720456b49339caaa9d313b6ce64 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 FOLLOWER]: Committing config change with OpId 1.73: config changed from index 32 to 73, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.265147 12026 consensus_queue.cc:237] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.74, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } }
I20240502 14:00:09.267318 13823 ts_tablet_manager.cc:1888] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.82
I20240502 14:00:09.267439 13823 log.cc:1196] T b9a6f7fca259430dac2ca17a8b180b2a P cbf92f56d7144a22a6d482c73c7c7ea9: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/wals/b9a6f7fca259430dac2ca17a8b180b2a
I20240502 14:00:09.268065 11613 catalog_manager.cc:4830] TS cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377): tablet b9a6f7fca259430dac2ca17a8b180b2a (table rebalance_test_table_1 [id=d6a2f7f1e4dc45ffbe7f9f2fba15577e]) successfully deleted
I20240502 14:00:09.268711 13433 raft_consensus.cc:1261] T c0230720456b49339caaa9d313b6ce64 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 FOLLOWER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch)
I20240502 14:00:09.269636 12808 consensus_queue.cc:1035] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20240502 14:00:09.271484 12808 raft_consensus.cc:2932] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Committing config change with OpId 1.75: config changed from index 73 to 75, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.273140 11612 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c0230720456b49339caaa9d313b6ce64 with cas_config_opid_index 73: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:09.273730 11919 tablet_copy_service.cc:140] P cbf92f56d7144a22a6d482c73c7c7ea9: Received BeginTabletCopySession request for tablet 236933482b6b41cf9e84a2eeb3b57638 from peer 7553d1bab6f642ac8395c928da1c58d7 ({username='slave'} at 127.8.86.135:35129)
I20240502 14:00:09.273631 13433 raft_consensus.cc:2932] T c0230720456b49339caaa9d313b6ce64 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 73 to 75, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.273826 11919 tablet_copy_service.cc:161] P cbf92f56d7144a22a6d482c73c7c7ea9: Beginning new tablet copy session on tablet 236933482b6b41cf9e84a2eeb3b57638 from peer 7553d1bab6f642ac8395c928da1c58d7 at {username='slave'} at 127.8.86.135:35129: session id = 7553d1bab6f642ac8395c928da1c58d7-236933482b6b41cf9e84a2eeb3b57638
I20240502 14:00:09.274785 11919 tablet_copy_source_session.cc:205] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:09.275686 11617 catalog_manager.cc:5484] T c0230720456b49339caaa9d313b6ce64 P 65a078666d30488f9c204d93e6394ef9 reported cstate change: config changed from index 73 to 75, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New cstate: current_term: 1 leader_uuid: "65a078666d30488f9c204d93e6394ef9" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:09.278934 13785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 236933482b6b41cf9e84a2eeb3b57638. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:09.280463 11747 tablet_service.cc:1514] Processing DeleteTablet for tablet c0230720456b49339caaa9d313b6ce64 with delete_type TABLET_DATA_TOMBSTONED (TS 73ce3d001cdf4d0c8e1487958b7694ea not found in new config with opid_index 75) from {username='slave'} at 127.0.0.1:43190
I20240502 14:00:09.281204 13876 tablet_replica.cc:330] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea: stopping tablet replica
I20240502 14:00:09.282322 13785 tablet_copy_client.cc:783] T 236933482b6b41cf9e84a2eeb3b57638 P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:09.282518 13785 tablet_copy_client.cc:647] T 236933482b6b41cf9e84a2eeb3b57638 P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:09.285615 13876 raft_consensus.cc:2229] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:09.285866 13876 raft_consensus.cc:2258] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:09.287988 13785 tablet_copy_client.cc:523] T 236933482b6b41cf9e84a2eeb3b57638 P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:09.289707 13785 tablet_bootstrap.cc:492] T 236933482b6b41cf9e84a2eeb3b57638 P 7553d1bab6f642ac8395c928da1c58d7: Bootstrap starting.
I20240502 14:00:09.306231 13718 tablet_bootstrap.cc:492] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28: Bootstrap replayed 1/1 log segments. Stats: ops{read=42 overwritten=0 applied=42 ignored=0} inserts{seen=673 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:09.306545 13718 tablet_bootstrap.cc:492] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28: Bootstrap complete.
I20240502 14:00:09.306717 13718 ts_tablet_manager.cc:1367] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28: Time spent bootstrapping tablet: real 0.324s user 0.050s sys 0.004s
I20240502 14:00:09.307109 13718 raft_consensus.cc:348] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } }
I20240502 14:00:09.307231 13718 raft_consensus.cc:729] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f6f2d2b1569415cbbbea2466dc28e28, State: Initialized, Role: LEARNER
I20240502 14:00:09.307318 13718 consensus_queue.cc:260] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 1.42, Last appended by leader: 42, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } }
I20240502 14:00:09.309231 13718 ts_tablet_manager.cc:1398] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:09.311242 12050 tablet_copy_service.cc:342] P 65a078666d30488f9c204d93e6394ef9: Request end of tablet copy session 4f6f2d2b1569415cbbbea2466dc28e28-1870fb0c459f49cab36812d16e30ddf5 received from {username='slave'} at 127.8.86.132:45333
I20240502 14:00:09.311353 12050 tablet_copy_service.cc:433] P 65a078666d30488f9c204d93e6394ef9: ending tablet copy session 4f6f2d2b1569415cbbbea2466dc28e28-1870fb0c459f49cab36812d16e30ddf5 on tablet 1870fb0c459f49cab36812d16e30ddf5 with peer 4f6f2d2b1569415cbbbea2466dc28e28
I20240502 14:00:09.311940 13876 ts_tablet_manager.cc:1875] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:09.319869 13876 ts_tablet_manager.cc:1888] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.74
I20240502 14:00:09.319995 13876 log.cc:1196] T c0230720456b49339caaa9d313b6ce64 P 73ce3d001cdf4d0c8e1487958b7694ea: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/wals/c0230720456b49339caaa9d313b6ce64
I20240502 14:00:09.320663 11612 catalog_manager.cc:4830] TS 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411): tablet c0230720456b49339caaa9d313b6ce64 (table rebalance_test_table_2 [id=d77f8a4d2b7c4f42901744c7061e0d56]) successfully deleted
I20240502 14:00:09.320785 13039 raft_consensus.cc:1203] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.39->[1.40-1.42] Dedup: 1.42->[]
I20240502 14:00:09.351277 13730 tablet_bootstrap.cc:492] T 259e249430fa41a2bc2528b65d022b67 P a36d5a88b1a34006bd3d040f5a715766: Bootstrap replayed 1/1 log segments. Stats: ops{read=62 overwritten=0 applied=61 ignored=0} inserts{seen=746 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:00:09.351716 13730 tablet_bootstrap.cc:492] T 259e249430fa41a2bc2528b65d022b67 P a36d5a88b1a34006bd3d040f5a715766: Bootstrap complete.
I20240502 14:00:09.351923 13730 ts_tablet_manager.cc:1367] T 259e249430fa41a2bc2528b65d022b67 P a36d5a88b1a34006bd3d040f5a715766: Time spent bootstrapping tablet: real 0.373s user 0.066s sys 0.007s
I20240502 14:00:09.352504 13730 raft_consensus.cc:348] T 259e249430fa41a2bc2528b65d022b67 P a36d5a88b1a34006bd3d040f5a715766 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } }
I20240502 14:00:09.353085 13730 raft_consensus.cc:729] T 259e249430fa41a2bc2528b65d022b67 P a36d5a88b1a34006bd3d040f5a715766 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a36d5a88b1a34006bd3d040f5a715766, State: Initialized, Role: LEARNER
I20240502 14:00:09.353226 13730 consensus_queue.cc:260] T 259e249430fa41a2bc2528b65d022b67 P a36d5a88b1a34006bd3d040f5a715766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 61, Last appended: 1.62, Last appended by leader: 62, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } }
I20240502 14:00:09.355325 13730 ts_tablet_manager.cc:1398] T 259e249430fa41a2bc2528b65d022b67 P a36d5a88b1a34006bd3d040f5a715766: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:00:09.357273 11788 tablet_copy_service.cc:342] P 73ce3d001cdf4d0c8e1487958b7694ea: Request end of tablet copy session a36d5a88b1a34006bd3d040f5a715766-259e249430fa41a2bc2528b65d022b67 received from {username='slave'} at 127.8.86.134:42887
I20240502 14:00:09.357374 11788 tablet_copy_service.cc:433] P 73ce3d001cdf4d0c8e1487958b7694ea: ending tablet copy session a36d5a88b1a34006bd3d040f5a715766-259e249430fa41a2bc2528b65d022b67 on tablet 259e249430fa41a2bc2528b65d022b67 with peer a36d5a88b1a34006bd3d040f5a715766
I20240502 14:00:09.405678 13785 tablet_bootstrap.cc:492] T 236933482b6b41cf9e84a2eeb3b57638 P 7553d1bab6f642ac8395c928da1c58d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=53 overwritten=0 applied=53 ignored=0} inserts{seen=738 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:09.406033 13785 tablet_bootstrap.cc:492] T 236933482b6b41cf9e84a2eeb3b57638 P 7553d1bab6f642ac8395c928da1c58d7: Bootstrap complete.
I20240502 14:00:09.406248 13785 ts_tablet_manager.cc:1367] T 236933482b6b41cf9e84a2eeb3b57638 P 7553d1bab6f642ac8395c928da1c58d7: Time spent bootstrapping tablet: real 0.117s user 0.050s sys 0.013s
I20240502 14:00:09.406782 13785 raft_consensus.cc:348] T 236933482b6b41cf9e84a2eeb3b57638 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } }
I20240502 14:00:09.406914 13785 raft_consensus.cc:729] T 236933482b6b41cf9e84a2eeb3b57638 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7553d1bab6f642ac8395c928da1c58d7, State: Initialized, Role: LEARNER
I20240502 14:00:09.407022 13785 consensus_queue.cc:260] T 236933482b6b41cf9e84a2eeb3b57638 P 7553d1bab6f642ac8395c928da1c58d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 53, Last appended: 1.53, Last appended by leader: 53, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } }
I20240502 14:00:09.407898 13785 ts_tablet_manager.cc:1398] T 236933482b6b41cf9e84a2eeb3b57638 P 7553d1bab6f642ac8395c928da1c58d7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:09.408692 11919 tablet_copy_service.cc:342] P cbf92f56d7144a22a6d482c73c7c7ea9: Request end of tablet copy session 7553d1bab6f642ac8395c928da1c58d7-236933482b6b41cf9e84a2eeb3b57638 received from {username='slave'} at 127.8.86.135:35129
I20240502 14:00:09.408784 11919 tablet_copy_service.cc:433] P cbf92f56d7144a22a6d482c73c7c7ea9: ending tablet copy session 7553d1bab6f642ac8395c928da1c58d7-236933482b6b41cf9e84a2eeb3b57638 on tablet 236933482b6b41cf9e84a2eeb3b57638 with peer 7553d1bab6f642ac8395c928da1c58d7
I20240502 14:00:09.445240 13785 ts_tablet_manager.cc:902] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7: Initiating tablet copy from peer 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 14:00:09.447376 13785 tablet_copy_client.cc:308] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:39411
I20240502 14:00:09.455921 11788 tablet_copy_service.cc:140] P 73ce3d001cdf4d0c8e1487958b7694ea: Received BeginTabletCopySession request for tablet b9bc1c84cb144965a21bdc40f099416d from peer 7553d1bab6f642ac8395c928da1c58d7 ({username='slave'} at 127.8.86.135:39205)
I20240502 14:00:09.456034 11788 tablet_copy_service.cc:161] P 73ce3d001cdf4d0c8e1487958b7694ea: Beginning new tablet copy session on tablet b9bc1c84cb144965a21bdc40f099416d from peer 7553d1bab6f642ac8395c928da1c58d7 at {username='slave'} at 127.8.86.135:39205: session id = 7553d1bab6f642ac8395c928da1c58d7-b9bc1c84cb144965a21bdc40f099416d
I20240502 14:00:09.457077 11788 tablet_copy_source_session.cc:205] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:09.460441 13785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9bc1c84cb144965a21bdc40f099416d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:09.462913 13785 tablet_copy_client.cc:783] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:09.463078 13785 tablet_copy_client.cc:647] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:09.479764 13785 tablet_copy_client.cc:523] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:09.481158 13785 tablet_bootstrap.cc:492] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7: Bootstrap starting.
I20240502 14:00:09.485299 13739 tablet_bootstrap.cc:492] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478: Bootstrap replayed 1/1 log segments. Stats: ops{read=89 overwritten=0 applied=89 ignored=0} inserts{seen=774 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:09.485630 13739 tablet_bootstrap.cc:492] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478: Bootstrap complete.
I20240502 14:00:09.485828 13739 ts_tablet_manager.cc:1367] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478: Time spent bootstrapping tablet: real 0.271s user 0.083s sys 0.010s
I20240502 14:00:09.486584 13739 raft_consensus.cc:348] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } }
I20240502 14:00:09.487144 13739 raft_consensus.cc:729] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5fbb448689c949d98dec8e37cf5c3478, State: Initialized, Role: LEARNER
I20240502 14:00:09.487273 13739 consensus_queue.cc:260] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 89, Last appended: 1.89, Last appended by leader: 89, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } }
I20240502 14:00:09.489238 13739 ts_tablet_manager.cc:1398] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:09.490146 11788 tablet_copy_service.cc:342] P 73ce3d001cdf4d0c8e1487958b7694ea: Request end of tablet copy session 5fbb448689c949d98dec8e37cf5c3478-3a50cea2712d48ca812cea7983d16840 received from {username='slave'} at 127.8.86.133:43393
I20240502 14:00:09.490226 11788 tablet_copy_service.cc:433] P 73ce3d001cdf4d0c8e1487958b7694ea: ending tablet copy session 5fbb448689c949d98dec8e37cf5c3478-3a50cea2712d48ca812cea7983d16840 on tablet 3a50cea2712d48ca812cea7983d16840 with peer 5fbb448689c949d98dec8e37cf5c3478
I20240502 14:00:09.508559 13433 raft_consensus.cc:1203] T 49041719e0ae42a390452b8716cc3ca7 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.63->[1.64-1.94] Dedup: 1.75->[1.76-1.94]
I20240502 14:00:09.532905 13303 raft_consensus.cc:1203] T 259e249430fa41a2bc2528b65d022b67 P a36d5a88b1a34006bd3d040f5a715766 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.45->[1.46-1.72] Dedup: 1.62->[1.63-1.72]
I20240502 14:00:09.573251 13617 raft_consensus.cc:1050] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9: attempting to promote NON_VOTER 4f6f2d2b1569415cbbbea2466dc28e28 to VOTER
I20240502 14:00:09.573658 13617 consensus_queue.cc:237] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 1.43, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } }
I20240502 14:00:09.574934 13038 raft_consensus.cc:1261] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 LEARNER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 1 index: 43. Preceding OpId from leader: term: 1 index: 44. (index mismatch)
I20240502 14:00:09.575752 13686 consensus_queue.cc:1035] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20240502 14:00:09.578694 11898 raft_consensus.cc:1261] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 1 index: 43. Preceding OpId from leader: term: 1 index: 44. (index mismatch)
I20240502 14:00:09.580577 13518 raft_consensus.cc:2932] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Committing config change with OpId 1.44: config changed from index 40 to 44, 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) changed from NON_VOTER to VOTER. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:09.582952 13039 raft_consensus.cc:2932] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Committing config change with OpId 1.44: config changed from index 40 to 44, 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) changed from NON_VOTER to VOTER. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:09.584059 11617 catalog_manager.cc:5484] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 reported cstate change: config changed from index 40 to 44, 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "65a078666d30488f9c204d93e6394ef9" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:09.584789 13686 consensus_queue.cc:1035] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20240502 14:00:09.587460 12026 consensus_queue.cc:237] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 1.44, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } }
I20240502 14:00:09.590449 11898 raft_consensus.cc:2932] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Committing config change with OpId 1.44: config changed from index 40 to 44, 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) changed from NON_VOTER to VOTER. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:09.591831 13040 raft_consensus.cc:1261] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 1 index: 44. Preceding OpId from leader: term: 1 index: 45. (index mismatch)
I20240502 14:00:09.592453 13617 consensus_queue.cc:1035] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20240502 14:00:09.594182 13686 raft_consensus.cc:2932] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Committing config change with OpId 1.45: config changed from index 44 to 45, VOTER cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) evicted. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:09.596340 11612 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1870fb0c459f49cab36812d16e30ddf5 with cas_config_opid_index 44: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:09.597393 13040 raft_consensus.cc:2932] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Committing config change with OpId 1.45: config changed from index 44 to 45, VOTER cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) evicted. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:09.597553 11617 catalog_manager.cc:5484] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 reported cstate change: config changed from index 44 to 45, VOTER cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) evicted. New cstate: current_term: 1 leader_uuid: "65a078666d30488f9c204d93e6394ef9" committed_config { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:09.600440 11877 tablet_service.cc:1514] Processing DeleteTablet for tablet 1870fb0c459f49cab36812d16e30ddf5 with delete_type TABLET_DATA_TOMBSTONED (TS cbf92f56d7144a22a6d482c73c7c7ea9 not found in new config with opid_index 45) from {username='slave'} at 127.0.0.1:37684
I20240502 14:00:09.600690 13823 tablet_replica.cc:330] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9: stopping tablet replica
I20240502 14:00:09.600833 13823 raft_consensus.cc:2229] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:09.600997 13823 raft_consensus.cc:2258] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:09.602893 13823 ts_tablet_manager.cc:1875] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:09.603777 13785 tablet_bootstrap.cc:492] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=65 overwritten=0 applied=65 ignored=0} inserts{seen=737 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:09.604072 13785 tablet_bootstrap.cc:492] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7: Bootstrap complete.
I20240502 14:00:09.604267 13785 ts_tablet_manager.cc:1367] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7: Time spent bootstrapping tablet: real 0.123s user 0.063s sys 0.011s
I20240502 14:00:09.604759 13785 raft_consensus.cc:348] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } }
I20240502 14:00:09.604912 13785 raft_consensus.cc:729] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7553d1bab6f642ac8395c928da1c58d7, State: Initialized, Role: LEARNER
I20240502 14:00:09.605031 13785 consensus_queue.cc:260] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 65, Last appended: 1.65, Last appended by leader: 65, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } }
I20240502 14:00:09.605192 13823 ts_tablet_manager.cc:1888] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.44
I20240502 14:00:09.605257 13823 log.cc:1196] T 1870fb0c459f49cab36812d16e30ddf5 P cbf92f56d7144a22a6d482c73c7c7ea9: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/wals/1870fb0c459f49cab36812d16e30ddf5
I20240502 14:00:09.605834 11613 catalog_manager.cc:4830] TS cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377): tablet 1870fb0c459f49cab36812d16e30ddf5 (table rebalance_test_table_3 [id=22735e7421b84776ac7e9ae79a0aa1f2]) successfully deleted
I20240502 14:00:09.605994 13785 ts_tablet_manager.cc:1398] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:00:09.606422 11788 tablet_copy_service.cc:342] P 73ce3d001cdf4d0c8e1487958b7694ea: Request end of tablet copy session 7553d1bab6f642ac8395c928da1c58d7-b9bc1c84cb144965a21bdc40f099416d received from {username='slave'} at 127.8.86.135:39205
I20240502 14:00:09.606527 11788 tablet_copy_service.cc:433] P 73ce3d001cdf4d0c8e1487958b7694ea: ending tablet copy session 7553d1bab6f642ac8395c928da1c58d7-b9bc1c84cb144965a21bdc40f099416d on tablet b9bc1c84cb144965a21bdc40f099416d with peer 7553d1bab6f642ac8395c928da1c58d7
I20240502 14:00:09.630807 13686 raft_consensus.cc:1050] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9: attempting to promote NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 to VOTER
I20240502 14:00:09.630791 13539 raft_consensus.cc:1050] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9: attempting to promote NON_VOTER 5fbb448689c949d98dec8e37cf5c3478 to VOTER
I20240502 14:00:09.631150 13686 consensus_queue.cc:237] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.95, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } }
I20240502 14:00:09.631176 13539 consensus_queue.cc:237] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } }
I20240502 14:00:09.632756 11767 raft_consensus.cc:1261] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240502 14:00:09.633571 13434 raft_consensus.cc:1261] T 49041719e0ae42a390452b8716cc3ca7 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 LEARNER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240502 14:00:09.634029 11767 raft_consensus.cc:1261] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:09.634670 13518 consensus_queue.cc:1035] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240502 14:00:09.634922 13518 consensus_queue.cc:1035] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 94, Time since last communication: 0.000s
I20240502 14:00:09.635455 13172 raft_consensus.cc:1261] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 1 LEARNER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:00:09.636718 13617 raft_consensus.cc:2932] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Committing config change with OpId 1.96: config changed from index 64 to 96, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.638491 11765 raft_consensus.cc:2932] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 64 to 96, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.640951 11617 catalog_manager.cc:5484] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 reported cstate change: config changed from index 64 to 96, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "65a078666d30488f9c204d93e6394ef9" committed_config { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:09.641557 13433 raft_consensus.cc:2932] T 49041719e0ae42a390452b8716cc3ca7 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 64 to 96, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.644243 13783 consensus_queue.cc:1035] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:09.645423 13783 consensus_queue.cc:1035] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:00:09.649102 12026 consensus_queue.cc:237] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 1.97, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } }
I20240502 14:00:09.649646 13661 raft_consensus.cc:2932] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
I20240502 14:00:09.650663 13434 raft_consensus.cc:1261] T 49041719e0ae42a390452b8716cc3ca7 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 FOLLOWER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
I20240502 14:00:09.651851 13617 consensus_queue.cc:1035] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 96, Time since last communication: 0.000s
I20240502 14:00:09.653193 11619 catalog_manager.cc:5484] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: config changed from index 2 to 3, 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:09.653424 13433 raft_consensus.cc:1203] T 49041719e0ae42a390452b8716cc3ca7 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.96->[1.97-1.98] Dedup: 1.97->[1.98-1.98]
I20240502 14:00:09.657021 13172 raft_consensus.cc:2932] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
I20240502 14:00:09.657591 11768 raft_consensus.cc:2932] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
I20240502 14:00:09.661888 12808 raft_consensus.cc:2932] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Committing config change with OpId 1.98: config changed from index 96 to 98, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.662634 13433 raft_consensus.cc:2932] T 49041719e0ae42a390452b8716cc3ca7 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index 96 to 98, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.664732 11617 catalog_manager.cc:5484] T 49041719e0ae42a390452b8716cc3ca7 P 65a078666d30488f9c204d93e6394ef9 reported cstate change: config changed from index 96 to 98, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New cstate: current_term: 1 leader_uuid: "65a078666d30488f9c204d93e6394ef9" committed_config { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:09.665176 11612 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49041719e0ae42a390452b8716cc3ca7 with cas_config_opid_index 96: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:09.667595 11747 tablet_service.cc:1514] Processing DeleteTablet for tablet 49041719e0ae42a390452b8716cc3ca7 with delete_type TABLET_DATA_TOMBSTONED (TS 73ce3d001cdf4d0c8e1487958b7694ea not found in new config with opid_index 98) from {username='slave'} at 127.0.0.1:43190
I20240502 14:00:09.667819 13876 tablet_replica.cc:330] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea: stopping tablet replica
I20240502 14:00:09.667964 13876 raft_consensus.cc:2229] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:09.668073 13876 pending_rounds.cc:62] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea: Trying to abort 1 pending ops.
I20240502 14:00:09.668138 13876 pending_rounds.cc:68] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea: Aborting op as it isn't in flight: id { term: 1 index: 97 } timestamp: 7023240845918134272 op_type: WRITE_OP write_request { tablet_id: "49041719e0ae42a390452b8716cc3ca7" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\242>\213$\366;\201s\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""098a53aab0f5c73c" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023240845855977472 authz_token { token_data: "\010\222\273\316\261\006\"3\n\005slave\022*\n 610fb42d93954513952217d0731890d9\020\001\030\001 \001(\001" signature: "<redacted>""Y\006T32\331ur>\246P\375VV\324?\327\210\332\336G\247\013<\t\336tN,\276\3252\354-V\264?\355:\031\202E\315\375\206\226\n\325\223.\003\220{\\\321,M\031|\311Nf\351\023" signing_key_seq_num: 0 } } request_id { client_id: "a5be2fb20e0147bb9374c03c081cb466" seq_no: 286 first_incomplete_seq_no: 286 attempt_no: 0 }
I20240502 14:00:09.668491 13876 raft_consensus.cc:2258] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:09.670001 13876 ts_tablet_manager.cc:1875] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:09.673297 13172 raft_consensus.cc:1203] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.79->[1.80-1.102] Dedup: 1.89->[1.90-1.102]
I20240502 14:00:09.683369 11899 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "03409b5f413f499c919f2c6a5a4126f7"
dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38678
I20240502 14:00:09.683540 11899 raft_consensus.cc:595] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:09.687777 13876 ts_tablet_manager.cc:1888] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.97
I20240502 14:00:09.687865 13876 log.cc:1196] T 49041719e0ae42a390452b8716cc3ca7 P 73ce3d001cdf4d0c8e1487958b7694ea: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/wals/49041719e0ae42a390452b8716cc3ca7
I20240502 14:00:09.688385 11612 catalog_manager.cc:4830] TS 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411): tablet 49041719e0ae42a390452b8716cc3ca7 (table rebalance_test_table_0 [id=610fb42d93954513952217d0731890d9]) successfully deleted
I20240502 14:00:09.698906 13569 raft_consensus.cc:1050] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea: attempting to promote NON_VOTER 5fbb448689c949d98dec8e37cf5c3478 to VOTER
I20240502 14:00:09.699240 13569 consensus_queue.cc:237] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } }
I20240502 14:00:09.702136 13172 raft_consensus.cc:1261] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 1 LEARNER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240502 14:00:09.703119 13506 consensus_queue.cc:1035] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 103, Time since last communication: 0.000s
I20240502 14:00:09.706741 13559 raft_consensus.cc:2932] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index 80 to 105, 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
I20240502 14:00:09.707163 13172 raft_consensus.cc:2932] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 80 to 105, 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
I20240502 14:00:09.709858 11619 catalog_manager.cc:5484] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 reported cstate change: config changed from index 80 to 105, 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
I20240502 14:00:09.712801 13433 raft_consensus.cc:1203] T 236933482b6b41cf9e84a2eeb3b57638 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.45->[1.46-1.56] Dedup: 1.53->[1.54-1.56]
I20240502 14:00:09.718452 11898 raft_consensus.cc:1261] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240502 14:00:09.722988 13569 consensus_queue.cc:1035] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 103, Time since last communication: 0.000s
I20240502 14:00:09.725369 11898 raft_consensus.cc:2932] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 80 to 105, 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
I20240502 14:00:09.728859 13783 raft_consensus.cc:1050] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9: attempting to promote NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 to VOTER
I20240502 14:00:09.729144 13783 consensus_queue.cc:237] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.57, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } }
I20240502 14:00:09.730985 11765 raft_consensus.cc:1261] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 58. (index mismatch)
I20240502 14:00:09.731205 13433 raft_consensus.cc:1261] T 236933482b6b41cf9e84a2eeb3b57638 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 LEARNER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 58. (index mismatch)
I20240502 14:00:09.732156 13661 consensus_queue.cc:1035] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20240502 14:00:09.733183 13661 consensus_queue.cc:1035] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20240502 14:00:09.734414 13661 raft_consensus.cc:2932] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Committing config change with OpId 1.58: config changed from index 46 to 58, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.734844 13433 raft_consensus.cc:2932] T 236933482b6b41cf9e84a2eeb3b57638 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 FOLLOWER]: Committing config change with OpId 1.58: config changed from index 46 to 58, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.737572 11617 catalog_manager.cc:5484] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: config changed from index 46 to 58, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:09.739006 11765 raft_consensus.cc:2932] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Committing config change with OpId 1.58: config changed from index 46 to 58, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.739076 11767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3a50cea2712d48ca812cea7983d16840"
dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53368
I20240502 14:00:09.740432 11767 raft_consensus.cc:595] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:09.742931 11898 consensus_queue.cc:237] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 1.59, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } }
I20240502 14:00:09.744094 13434 raft_consensus.cc:1261] T 236933482b6b41cf9e84a2eeb3b57638 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch)
I20240502 14:00:09.745435 13661 consensus_queue.cc:1035] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20240502 14:00:09.746583 13569 raft_consensus.cc:979] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea: : Instructing follower cbf92f56d7144a22a6d482c73c7c7ea9 to start an election
I20240502 14:00:09.746668 13569 raft_consensus.cc:1067] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Signalling peer cbf92f56d7144a22a6d482c73c7c7ea9 to start an election
I20240502 14:00:09.747213 11898 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3a50cea2712d48ca812cea7983d16840"
dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9"
from {username='slave'} at 127.8.86.129:51843
I20240502 14:00:09.747133 13572 raft_consensus.cc:2932] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Committing config change with OpId 1.60: config changed from index 58 to 60, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.747335 11898 raft_consensus.cc:482] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:09.747392 11898 raft_consensus.cc:3037] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:09.748389 11898 raft_consensus.cc:504] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } }
I20240502 14:00:09.748759 11898 leader_election.cc:288] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 2 election: Requested vote from peers 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411), 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133:44919)
I20240502 14:00:09.749336 13434 raft_consensus.cc:2932] T 236933482b6b41cf9e84a2eeb3b57638 P 7553d1bab6f642ac8395c928da1c58d7 [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index 58 to 60, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.749475 11613 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 236933482b6b41cf9e84a2eeb3b57638 with cas_config_opid_index 58: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:09.749933 11767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a50cea2712d48ca812cea7983d16840" candidate_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: true dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
I20240502 14:00:09.750051 11767 raft_consensus.cc:3032] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:09.750102 11767 raft_consensus.cc:729] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Running, Role: LEADER
I20240502 14:00:09.750255 11767 consensus_queue.cc:260] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 105, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } }
I20240502 14:00:09.750440 11767 raft_consensus.cc:3037] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:09.750582 11617 catalog_manager.cc:5484] T 236933482b6b41cf9e84a2eeb3b57638 P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: config changed from index 58 to 60, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New cstate: current_term: 1 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:09.751147 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a50cea2712d48ca812cea7983d16840" candidate_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: true dest_uuid: "5fbb448689c949d98dec8e37cf5c3478"
I20240502 14:00:09.751343 13172 raft_consensus.cc:3037] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:09.751443 11767 raft_consensus.cc:2454] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbf92f56d7144a22a6d482c73c7c7ea9 in term 2.
I20240502 14:00:09.751721 11834 leader_election.cc:302] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73ce3d001cdf4d0c8e1487958b7694ea, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:09.752213 13661 raft_consensus.cc:2781] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:09.752310 13661 raft_consensus.cc:686] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 LEADER]: Becoming Leader. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Running, Role: LEADER
I20240502 14:00:09.752435 13172 raft_consensus.cc:2454] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbf92f56d7144a22a6d482c73c7c7ea9 in term 2.
I20240502 14:00:09.752436 13661 consensus_queue.cc:237] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } }
W20240502 14:00:09.755687 11727 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53188: Illegal state: replica 73ce3d001cdf4d0c8e1487958b7694ea is not leader of this config: current role FOLLOWER
I20240502 14:00:09.762708 13172 raft_consensus.cc:1261] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 2 index: 107. (index mismatch)
I20240502 14:00:09.762650 11767 raft_consensus.cc:1261] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 2 index: 107. (index mismatch)
I20240502 14:00:09.763315 13783 consensus_queue.cc:1035] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20240502 14:00:09.763612 13783 consensus_queue.cc:1035] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20240502 14:00:09.763921 11747 tablet_service.cc:1514] Processing DeleteTablet for tablet 236933482b6b41cf9e84a2eeb3b57638 with delete_type TABLET_DATA_TOMBSTONED (TS 73ce3d001cdf4d0c8e1487958b7694ea not found in new config with opid_index 60) from {username='slave'} at 127.0.0.1:43190
I20240502 14:00:09.764117 13876 tablet_replica.cc:330] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea: stopping tablet replica
I20240502 14:00:09.764248 13876 raft_consensus.cc:2229] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:09.764365 13876 raft_consensus.cc:2258] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:09.765130 13876 ts_tablet_manager.cc:1875] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:09.765198 11617 catalog_manager.cc:5484] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: term changed from 1 to 2, leader changed from 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) to cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130). New cstate: current_term: 2 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:09.767310 13876 ts_tablet_manager.cc:1888] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.59
I20240502 14:00:09.767390 13876 log.cc:1196] T 236933482b6b41cf9e84a2eeb3b57638 P 73ce3d001cdf4d0c8e1487958b7694ea: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/wals/236933482b6b41cf9e84a2eeb3b57638
I20240502 14:00:09.768355 11612 catalog_manager.cc:4830] TS 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411): tablet 236933482b6b41cf9e84a2eeb3b57638 (table rebalance_test_table_3 [id=22735e7421b84776ac7e9ae79a0aa1f2]) successfully deleted
I20240502 14:00:09.773968 13791 mvcc.cc:205] Tried to move back new op lower bound from 7023240846383542272 to 7023240846351163392. Current Snapshot: MvccSnapshot[applied={T|T < 7023240846383542272}]
I20240502 14:00:09.778121 11898 consensus_queue.cc:237] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 2.107, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } }
I20240502 14:00:09.781589 13172 raft_consensus.cc:1261] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 2 index: 107. Preceding OpId from leader: term: 2 index: 108. (index mismatch)
I20240502 14:00:09.781967 13783 consensus_queue.cc:1035] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20240502 14:00:09.788105 13539 raft_consensus.cc:2932] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 LEADER]: Committing config change with OpId 2.108: config changed from index 105 to 108, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
I20240502 14:00:09.789731 11613 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3a50cea2712d48ca812cea7983d16840 with cas_config_opid_index 105: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:09.795142 11617 catalog_manager.cc:5484] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: config changed from index 105 to 108, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New cstate: current_term: 2 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:09.796705 13172 raft_consensus.cc:2932] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Committing config change with OpId 2.108: config changed from index 105 to 108, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
I20240502 14:00:09.798365 11747 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a50cea2712d48ca812cea7983d16840 with delete_type TABLET_DATA_TOMBSTONED (TS 73ce3d001cdf4d0c8e1487958b7694ea not found in new config with opid_index 108) from {username='slave'} at 127.0.0.1:43190
I20240502 14:00:09.799093 13876 tablet_replica.cc:330] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea: stopping tablet replica
I20240502 14:00:09.799222 13876 raft_consensus.cc:2229] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:09.799337 13876 raft_consensus.cc:2258] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:09.801239 13876 ts_tablet_manager.cc:1875] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:09.807288 13876 ts_tablet_manager.cc:1888] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.107
I20240502 14:00:09.807399 13876 log.cc:1196] T 3a50cea2712d48ca812cea7983d16840 P 73ce3d001cdf4d0c8e1487958b7694ea: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/wals/3a50cea2712d48ca812cea7983d16840
I20240502 14:00:09.835422 13434 raft_consensus.cc:1203] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.55->[1.56-1.65] Dedup: 1.65->[]
I20240502 14:00:09.840073 11612 catalog_manager.cc:4830] TS 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411): tablet 3a50cea2712d48ca812cea7983d16840 (table rebalance_test_table_1 [id=d6a2f7f1e4dc45ffbe7f9f2fba15577e]) successfully deleted
I20240502 14:00:09.899461 13699 raft_consensus.cc:1050] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea: attempting to promote NON_VOTER a36d5a88b1a34006bd3d040f5a715766 to VOTER
I20240502 14:00:09.899837 13569 raft_consensus.cc:1050] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea: attempting to promote NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 to VOTER
I20240502 14:00:09.899791 13699 consensus_queue.cc:237] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } }
I20240502 14:00:09.900084 13569 consensus_queue.cc:237] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 1.67, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } }
I20240502 14:00:09.901156 12026 raft_consensus.cc:1261] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240502 14:00:09.901437 12027 raft_consensus.cc:1261] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch)
I20240502 14:00:09.901978 13506 consensus_queue.cc:1035] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 72, Time since last communication: 0.000s
I20240502 14:00:09.902124 13559 consensus_queue.cc:1035] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20240502 14:00:09.904053 13559 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Committing config change with OpId 1.68: config changed from index 56 to 68, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.904561 12030 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index 56 to 68, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.907248 13302 raft_consensus.cc:1261] T 259e249430fa41a2bc2528b65d022b67 P a36d5a88b1a34006bd3d040f5a715766 [term 1 LEARNER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240502 14:00:09.907415 11619 catalog_manager.cc:5484] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9 reported cstate change: config changed from index 56 to 68, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.907727 13506 raft_consensus.cc:2932] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index 46 to 74, a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:09.909828 13700 consensus_queue.cc:1035] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240502 14:00:09.911593 13302 raft_consensus.cc:2932] T 259e249430fa41a2bc2528b65d022b67 P a36d5a88b1a34006bd3d040f5a715766 [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index 46 to 74, a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:09.913791 13434 raft_consensus.cc:1261] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 1 LEARNER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch)
I20240502 14:00:09.913971 11617 catalog_manager.cc:5484] T 259e249430fa41a2bc2528b65d022b67 P a36d5a88b1a34006bd3d040f5a715766 reported cstate change: config changed from index 46 to 74, a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:09.914405 13566 consensus_queue.cc:1035] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20240502 14:00:09.915339 13434 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index 56 to 68, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.920939 12027 raft_consensus.cc:2932] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index 46 to 74, a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:09.940008 11767 consensus_queue.cc:237] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 1.75, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } }
I20240502 14:00:09.941210 13303 raft_consensus.cc:1261] T 259e249430fa41a2bc2528b65d022b67 P a36d5a88b1a34006bd3d040f5a715766 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240502 14:00:09.945673 13699 consensus_queue.cc:1035] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20240502 14:00:09.947517 11765 consensus_queue.cc:237] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } }
I20240502 14:00:09.949394 13434 raft_consensus.cc:1261] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240502 14:00:09.949959 13699 consensus_queue.cc:1035] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 68, Time since last communication: 0.000s
I20240502 14:00:09.950119 13700 raft_consensus.cc:2932] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Committing config change with OpId 1.76: config changed from index 74 to 76, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:09.952891 11612 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 259e249430fa41a2bc2528b65d022b67 with cas_config_opid_index 74: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:09.953841 11617 catalog_manager.cc:5484] T 259e249430fa41a2bc2528b65d022b67 P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: config changed from index 74 to 76, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:09.957531 12008 tablet_service.cc:1514] Processing DeleteTablet for tablet 259e249430fa41a2bc2528b65d022b67 with delete_type TABLET_DATA_TOMBSTONED (TS 65a078666d30488f9c204d93e6394ef9 not found in new config with opid_index 76) from {username='slave'} at 127.0.0.1:38592
I20240502 14:00:09.960824 13968 tablet_replica.cc:330] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9: stopping tablet replica
I20240502 14:00:09.961027 13968 raft_consensus.cc:2229] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:09.962617 13302 raft_consensus.cc:2932] T 259e249430fa41a2bc2528b65d022b67 P a36d5a88b1a34006bd3d040f5a715766 [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index 74 to 76, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:09.961218 13968 raft_consensus.cc:2258] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:09.964653 13700 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index 68 to 70, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.966181 11612 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b9bc1c84cb144965a21bdc40f099416d with cas_config_opid_index 68: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:09.967280 11617 catalog_manager.cc:5484] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: config changed from index 68 to 70, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:09.968740 13433 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index 68 to 70, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:09.968858 13968 ts_tablet_manager.cc:1875] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:09.973495 12008 tablet_service.cc:1514] Processing DeleteTablet for tablet b9bc1c84cb144965a21bdc40f099416d with delete_type TABLET_DATA_TOMBSTONED (TS 65a078666d30488f9c204d93e6394ef9 not found in new config with opid_index 70) from {username='slave'} at 127.0.0.1:38592
I20240502 14:00:09.977550 13968 ts_tablet_manager.cc:1888] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.75
I20240502 14:00:09.977741 13968 log.cc:1196] T 259e249430fa41a2bc2528b65d022b67 P 65a078666d30488f9c204d93e6394ef9: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/wals/259e249430fa41a2bc2528b65d022b67
I20240502 14:00:09.996834 13968 tablet_replica.cc:330] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9: stopping tablet replica
I20240502 14:00:09.997052 13968 raft_consensus.cc:2229] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:09.997488 13968 raft_consensus.cc:2258] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:09.997598 11612 catalog_manager.cc:4830] TS 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325): tablet 259e249430fa41a2bc2528b65d022b67 (table rebalance_test_table_4 [id=83b2409333254a509079ef88e0c156f9]) successfully deleted
I20240502 14:00:09.998476 13968 ts_tablet_manager.cc:1875] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:10.007973 13968 ts_tablet_manager.cc:1888] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.69
I20240502 14:00:10.008061 13968 log.cc:1196] T b9bc1c84cb144965a21bdc40f099416d P 65a078666d30488f9c204d93e6394ef9: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/wals/b9bc1c84cb144965a21bdc40f099416d
I20240502 14:00:10.008623 11612 catalog_manager.cc:4830] TS 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325): tablet b9bc1c84cb144965a21bdc40f099416d (table rebalance_test_table_4 [id=83b2409333254a509079ef88e0c156f9]) successfully deleted
I20240502 14:00:10.064393 13783 raft_consensus.cc:979] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9: : Instructing follower 73ce3d001cdf4d0c8e1487958b7694ea to start an election
I20240502 14:00:10.064527 13783 raft_consensus.cc:1067] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Signalling peer 73ce3d001cdf4d0c8e1487958b7694ea to start an election
I20240502 14:00:10.065135 11765 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "03409b5f413f499c919f2c6a5a4126f7"
dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
from {username='slave'} at 127.8.86.130:32929
I20240502 14:00:10.065294 11765 raft_consensus.cc:482] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:10.065353 11765 raft_consensus.cc:3037] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:10.066354 11765 raft_consensus.cc:504] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } }
I20240502 14:00:10.066759 11765 leader_election.cc:288] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 2 election: Requested vote from peers cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377), 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133:44919)
I20240502 14:00:10.068099 11898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03409b5f413f499c919f2c6a5a4126f7" candidate_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9"
I20240502 14:00:10.068219 11898 raft_consensus.cc:3032] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:10.068284 11898 raft_consensus.cc:729] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Running, Role: LEADER
I20240502 14:00:10.068434 11898 consensus_queue.cc:260] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } }
I20240502 14:00:10.068584 11898 raft_consensus.cc:3037] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:10.069617 11898 raft_consensus.cc:2454] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73ce3d001cdf4d0c8e1487958b7694ea in term 2.
I20240502 14:00:10.069927 11704 leader_election.cc:302] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73ce3d001cdf4d0c8e1487958b7694ea, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:10.070086 13559 raft_consensus.cc:2781] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:10.070156 13559 raft_consensus.cc:686] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 LEADER]: Becoming Leader. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Running, Role: LEADER
I20240502 14:00:10.070348 13559 consensus_queue.cc:237] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } }
I20240502 14:00:10.072232 13170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03409b5f413f499c919f2c6a5a4126f7" candidate_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5fbb448689c949d98dec8e37cf5c3478"
I20240502 14:00:10.072360 13170 raft_consensus.cc:3037] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:10.072914 11619 catalog_manager.cc:5484] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: term changed from 1 to 2, leader changed from cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) to 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129). New cstate: current_term: 2 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:10.073343 13170 raft_consensus.cc:2454] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73ce3d001cdf4d0c8e1487958b7694ea in term 2.
I20240502 14:00:10.510605 13170 raft_consensus.cc:1261] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:00:10.511528 13559 consensus_queue.cc:1035] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:10.515422 11898 raft_consensus.cc:1261] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:00:10.516456 13569 consensus_queue.cc:1035] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:10.520177 11767 consensus_queue.cc:237] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } }
I20240502 14:00:10.522516 13170 raft_consensus.cc:1261] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:00:10.523618 13506 consensus_queue.cc:1035] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:10.527961 13506 raft_consensus.cc:2932] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
I20240502 14:00:10.529683 13170 raft_consensus.cc:2932] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
I20240502 14:00:10.531561 11612 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 03409b5f413f499c919f2c6a5a4126f7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:10.533612 11619 catalog_manager.cc:5484] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: config changed from index 3 to 5, VOTER cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) evicted. New cstate: current_term: 2 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:10.540735 11877 tablet_service.cc:1514] Processing DeleteTablet for tablet 03409b5f413f499c919f2c6a5a4126f7 with delete_type TABLET_DATA_TOMBSTONED (TS cbf92f56d7144a22a6d482c73c7c7ea9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37684
I20240502 14:00:10.542347 14019 tablet_replica.cc:330] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9: stopping tablet replica
I20240502 14:00:10.542497 14019 raft_consensus.cc:2229] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:10.542619 14019 raft_consensus.cc:2258] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:10.543543 14019 ts_tablet_manager.cc:1875] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:10.548322 14019 ts_tablet_manager.cc:1888] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 14:00:10.548400 14019 log.cc:1196] T 03409b5f413f499c919f2c6a5a4126f7 P cbf92f56d7144a22a6d482c73c7c7ea9: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/wals/03409b5f413f499c919f2c6a5a4126f7
I20240502 14:00:10.553709 11613 catalog_manager.cc:4830] TS cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377): tablet 03409b5f413f499c919f2c6a5a4126f7 (table TestTable [id=977ef0b76afd4ccc955a0f354e4f5725]) successfully deleted
I20240502 14:00:10.880214 13414 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:10.890794 13020 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:10.896234 13283 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:10.897215 11877 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:10.899927 13151 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:10.900458 12008 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:10.907871 11747 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:11.001519 11898 consensus_queue.cc:237] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 1.112, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } }
I20240502 14:00:11.003219 11765 raft_consensus.cc:1261] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 112. Preceding OpId from leader: term: 1 index: 113. (index mismatch)
I20240502 14:00:11.004081 13572 consensus_queue.cc:1035] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20240502 14:00:11.009040 13539 raft_consensus.cc:2932] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Committing config change with OpId 1.113: config changed from index -1 to 113, NON_VOTER a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) added. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } } }
I20240502 14:00:11.009658 11767 raft_consensus.cc:2932] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Committing config change with OpId 1.113: config changed from index -1 to 113, NON_VOTER a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) added. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } } }
W20240502 14:00:11.014014 11832 consensus_peers.cc:485] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 -> Peer a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134:36541): Couldn't send request to peer a36d5a88b1a34006bd3d040f5a715766. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 182302167a0541888cb99bcd66635125. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:11.014532 11624 catalog_manager.cc:5484] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: config changed from index -1 to 113, NON_VOTER a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:11.036710 11767 consensus_queue.cc:237] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 1.112, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } }
W20240502 14:00:11.038362 11703 consensus_peers.cc:485] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea -> Peer 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133:44919): Couldn't send request to peer 5fbb448689c949d98dec8e37cf5c3478. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9bc1c84cb144965a21bdc40f099416d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:11.038743 13434 raft_consensus.cc:1261] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 112. Preceding OpId from leader: term: 1 index: 113. (index mismatch)
I20240502 14:00:11.039120 13569 consensus_queue.cc:1035] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20240502 14:00:11.040979 13559 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Committing config change with OpId 1.113: config changed from index 70 to 113, NON_VOTER 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) added. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } } }
I20240502 14:00:11.041271 13434 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 1 FOLLOWER]: Committing config change with OpId 1.113: config changed from index 70 to 113, NON_VOTER 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) added. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } } }
I20240502 14:00:11.044189 11622 catalog_manager.cc:5484] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 reported cstate change: config changed from index 70 to 113, NON_VOTER 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } } }
I20240502 14:00:11.060863 12027 consensus_queue.cc:237] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 153, Committed index: 153, Last appended: 1.153, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } }
W20240502 14:00:11.062739 11966 consensus_peers.cc:485] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 -> Peer 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132:42709): Couldn't send request to peer 4f6f2d2b1569415cbbbea2466dc28e28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db4ce5e277504649a781e23861baa667. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:11.063056 11898 raft_consensus.cc:1261] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 1 index: 153. Preceding OpId from leader: term: 1 index: 154. (index mismatch)
I20240502 14:00:11.064433 13686 consensus_queue.cc:1035] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 154, Last known committed idx: 153, Time since last communication: 0.000s
I20240502 14:00:11.067464 13617 raft_consensus.cc:2932] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Committing config change with OpId 1.154: config changed from index -1 to 154, NON_VOTER 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) added. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } } }
I20240502 14:00:11.067894 11898 raft_consensus.cc:2932] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Committing config change with OpId 1.154: config changed from index -1 to 154, NON_VOTER 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) added. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } } }
I20240502 14:00:11.070500 11622 catalog_manager.cc:5484] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: config changed from index -1 to 154, NON_VOTER 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) added. New cstate: current_term: 1 leader_uuid: "65a078666d30488f9c204d93e6394ef9" committed_config { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } } }
I20240502 14:00:11.097885 11765 consensus_queue.cc:237] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } }
I20240502 14:00:11.099063 13172 raft_consensus.cc:1261] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:00:11.099679 13559 consensus_queue.cc:1035] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:00:11.101651 11704 consensus_peers.cc:485] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea -> Peer 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135:34019): Couldn't send request to peer 7553d1bab6f642ac8395c928da1c58d7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03409b5f413f499c919f2c6a5a4126f7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:11.103150 13559 raft_consensus.cc:2932] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } } }
I20240502 14:00:11.105279 13170 raft_consensus.cc:2932] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } } }
I20240502 14:00:11.108198 11624 catalog_manager.cc:5484] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: config changed from index 5 to 6, NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) added. New cstate: current_term: 2 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:11.137300 11898 consensus_queue.cc:237] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 1.119, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } }
I20240502 14:00:11.138969 12027 raft_consensus.cc:1261] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 119. Preceding OpId from leader: term: 1 index: 120. (index mismatch)
W20240502 14:00:11.141165 11832 consensus_peers.cc:485] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 -> Peer a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134:36541): Couldn't send request to peer a36d5a88b1a34006bd3d040f5a715766. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 986b926ead014774925e3b8a3892c56d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:11.142141 13661 consensus_queue.cc:1035] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 119, Time since last communication: 0.000s
I20240502 14:00:11.145742 13572 raft_consensus.cc:2932] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Committing config change with OpId 1.120: config changed from index -1 to 120, NON_VOTER a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) added. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } } }
I20240502 14:00:11.146660 12030 raft_consensus.cc:2932] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Committing config change with OpId 1.120: config changed from index -1 to 120, NON_VOTER a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) added. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } } }
I20240502 14:00:11.148907 11624 catalog_manager.cc:5484] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: config changed from index -1 to 120, NON_VOTER a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:11.465806 14137 ts_tablet_manager.cc:902] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28: Initiating tablet copy from peer 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:11.466414 14137 tablet_copy_client.cc:308] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:45325
I20240502 14:00:11.470531 12050 tablet_copy_service.cc:140] P 65a078666d30488f9c204d93e6394ef9: Received BeginTabletCopySession request for tablet db4ce5e277504649a781e23861baa667 from peer 4f6f2d2b1569415cbbbea2466dc28e28 ({username='slave'} at 127.8.86.132:45333)
I20240502 14:00:11.470646 12050 tablet_copy_service.cc:161] P 65a078666d30488f9c204d93e6394ef9: Beginning new tablet copy session on tablet db4ce5e277504649a781e23861baa667 from peer 4f6f2d2b1569415cbbbea2466dc28e28 at {username='slave'} at 127.8.86.132:45333: session id = 4f6f2d2b1569415cbbbea2466dc28e28-db4ce5e277504649a781e23861baa667
I20240502 14:00:11.471486 12050 tablet_copy_source_session.cc:205] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:11.476114 14137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4ce5e277504649a781e23861baa667. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:11.479146 14137 tablet_copy_client.cc:783] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:11.479312 14137 tablet_copy_client.cc:647] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:11.483973 14137 tablet_copy_client.cc:523] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:11.485354 14137 tablet_bootstrap.cc:492] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28: Bootstrap starting.
I20240502 14:00:11.520488 14150 ts_tablet_manager.cc:902] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766: Initiating tablet copy from peer cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377)
I20240502 14:00:11.521497 14150 tablet_copy_client.cc:308] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:44377
I20240502 14:00:11.522523 14149 ts_tablet_manager.cc:902] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7: Initiating tablet copy from peer 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 14:00:11.523144 14149 tablet_copy_client.cc:308] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:39411
I20240502 14:00:11.523687 11788 tablet_copy_service.cc:140] P 73ce3d001cdf4d0c8e1487958b7694ea: Received BeginTabletCopySession request for tablet 03409b5f413f499c919f2c6a5a4126f7 from peer 7553d1bab6f642ac8395c928da1c58d7 ({username='slave'} at 127.8.86.135:39205)
I20240502 14:00:11.523793 11788 tablet_copy_service.cc:161] P 73ce3d001cdf4d0c8e1487958b7694ea: Beginning new tablet copy session on tablet 03409b5f413f499c919f2c6a5a4126f7 from peer 7553d1bab6f642ac8395c928da1c58d7 at {username='slave'} at 127.8.86.135:39205: session id = 7553d1bab6f642ac8395c928da1c58d7-03409b5f413f499c919f2c6a5a4126f7
I20240502 14:00:11.524758 11788 tablet_copy_source_session.cc:205] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:11.525915 14149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03409b5f413f499c919f2c6a5a4126f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:11.528563 14149 tablet_copy_client.cc:783] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:11.528774 14149 tablet_copy_client.cc:647] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:11.530187 14149 tablet_copy_client.cc:523] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:11.531761 14149 tablet_bootstrap.cc:492] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7: Bootstrap starting.
I20240502 14:00:11.540320 14149 tablet_bootstrap.cc:492] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:11.542589 14149 tablet_bootstrap.cc:492] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7: Bootstrap complete.
I20240502 14:00:11.542835 14149 ts_tablet_manager.cc:1367] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20240502 14:00:11.543344 14149 raft_consensus.cc:348] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } }
I20240502 14:00:11.543516 14149 raft_consensus.cc:729] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7553d1bab6f642ac8395c928da1c58d7, State: Initialized, Role: LEARNER
I20240502 14:00:11.543655 14149 consensus_queue.cc:260] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: true } }
I20240502 14:00:11.544063 14149 ts_tablet_manager.cc:1398] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:11.545287 11788 tablet_copy_service.cc:342] P 73ce3d001cdf4d0c8e1487958b7694ea: Request end of tablet copy session 7553d1bab6f642ac8395c928da1c58d7-03409b5f413f499c919f2c6a5a4126f7 received from {username='slave'} at 127.8.86.135:39205
I20240502 14:00:11.545374 11788 tablet_copy_service.cc:433] P 73ce3d001cdf4d0c8e1487958b7694ea: ending tablet copy session 7553d1bab6f642ac8395c928da1c58d7-03409b5f413f499c919f2c6a5a4126f7 on tablet 03409b5f413f499c919f2c6a5a4126f7 with peer 7553d1bab6f642ac8395c928da1c58d7
I20240502 14:00:11.605537 11919 tablet_copy_service.cc:140] P cbf92f56d7144a22a6d482c73c7c7ea9: Received BeginTabletCopySession request for tablet 182302167a0541888cb99bcd66635125 from peer a36d5a88b1a34006bd3d040f5a715766 ({username='slave'} at 127.8.86.134:45157)
I20240502 14:00:11.605679 11919 tablet_copy_service.cc:161] P cbf92f56d7144a22a6d482c73c7c7ea9: Beginning new tablet copy session on tablet 182302167a0541888cb99bcd66635125 from peer a36d5a88b1a34006bd3d040f5a715766 at {username='slave'} at 127.8.86.134:45157: session id = a36d5a88b1a34006bd3d040f5a715766-182302167a0541888cb99bcd66635125
I20240502 14:00:11.606624 11919 tablet_copy_source_session.cc:205] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:11.607586 14150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182302167a0541888cb99bcd66635125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:11.610105 14150 tablet_copy_client.cc:783] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:11.610280 14150 tablet_copy_client.cc:647] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:11.621407 14173 ts_tablet_manager.cc:902] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766: Initiating tablet copy from peer cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377)
I20240502 14:00:11.622143 14173 tablet_copy_client.cc:308] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:44377
I20240502 14:00:11.622668 11919 tablet_copy_service.cc:140] P cbf92f56d7144a22a6d482c73c7c7ea9: Received BeginTabletCopySession request for tablet 986b926ead014774925e3b8a3892c56d from peer a36d5a88b1a34006bd3d040f5a715766 ({username='slave'} at 127.8.86.134:45157)
I20240502 14:00:11.622781 11919 tablet_copy_service.cc:161] P cbf92f56d7144a22a6d482c73c7c7ea9: Beginning new tablet copy session on tablet 986b926ead014774925e3b8a3892c56d from peer a36d5a88b1a34006bd3d040f5a715766 at {username='slave'} at 127.8.86.134:45157: session id = a36d5a88b1a34006bd3d040f5a715766-986b926ead014774925e3b8a3892c56d
I20240502 14:00:11.623483 11919 tablet_copy_source_session.cc:205] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:11.624266 14173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986b926ead014774925e3b8a3892c56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:11.626466 14173 tablet_copy_client.cc:783] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:11.626614 14173 tablet_copy_client.cc:647] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:11.628312 14173 tablet_copy_client.cc:523] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:11.628561 14150 tablet_copy_client.cc:523] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:11.631871 14150 tablet_bootstrap.cc:492] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766: Bootstrap starting.
I20240502 14:00:11.646831 14173 tablet_bootstrap.cc:492] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766: Bootstrap starting.
I20240502 14:00:11.675513 14178 ts_tablet_manager.cc:902] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478: Initiating tablet copy from peer 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 14:00:11.676158 14178 tablet_copy_client.cc:308] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:39411
I20240502 14:00:11.678413 11788 tablet_copy_service.cc:140] P 73ce3d001cdf4d0c8e1487958b7694ea: Received BeginTabletCopySession request for tablet b9bc1c84cb144965a21bdc40f099416d from peer 5fbb448689c949d98dec8e37cf5c3478 ({username='slave'} at 127.8.86.133:43393)
I20240502 14:00:11.678522 11788 tablet_copy_service.cc:161] P 73ce3d001cdf4d0c8e1487958b7694ea: Beginning new tablet copy session on tablet b9bc1c84cb144965a21bdc40f099416d from peer 5fbb448689c949d98dec8e37cf5c3478 at {username='slave'} at 127.8.86.133:43393: session id = 5fbb448689c949d98dec8e37cf5c3478-b9bc1c84cb144965a21bdc40f099416d
I20240502 14:00:11.679536 11788 tablet_copy_source_session.cc:205] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:11.679544 14137 tablet_bootstrap.cc:492] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28: Bootstrap replayed 1/1 log segments. Stats: ops{read=159 overwritten=0 applied=158 ignored=0} inserts{seen=789 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:00:11.679951 14137 tablet_bootstrap.cc:492] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28: Bootstrap complete.
I20240502 14:00:11.680516 14178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9bc1c84cb144965a21bdc40f099416d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:11.682907 14178 tablet_copy_client.cc:783] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:11.683084 14178 tablet_copy_client.cc:647] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:11.683379 14137 ts_tablet_manager.cc:1367] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28: Time spent bootstrapping tablet: real 0.198s user 0.130s sys 0.014s
I20240502 14:00:11.683863 14137 raft_consensus.cc:348] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } }
I20240502 14:00:11.684175 14137 raft_consensus.cc:729] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f6f2d2b1569415cbbbea2466dc28e28, State: Initialized, Role: LEARNER
I20240502 14:00:11.684293 14137 consensus_queue.cc:260] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 158, Last appended: 1.159, Last appended by leader: 159, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: true } }
I20240502 14:00:11.684715 14137 ts_tablet_manager.cc:1398] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:11.687244 12050 tablet_copy_service.cc:342] P 65a078666d30488f9c204d93e6394ef9: Request end of tablet copy session 4f6f2d2b1569415cbbbea2466dc28e28-db4ce5e277504649a781e23861baa667 received from {username='slave'} at 127.8.86.132:45333
I20240502 14:00:11.687336 12050 tablet_copy_service.cc:433] P 65a078666d30488f9c204d93e6394ef9: ending tablet copy session 4f6f2d2b1569415cbbbea2466dc28e28-db4ce5e277504649a781e23861baa667 on tablet db4ce5e277504649a781e23861baa667 with peer 4f6f2d2b1569415cbbbea2466dc28e28
I20240502 14:00:11.695546 14178 tablet_copy_client.cc:523] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:11.696991 14178 tablet_bootstrap.cc:492] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478: Bootstrap starting.
I20240502 14:00:11.926589 14178 tablet_bootstrap.cc:492] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478: Bootstrap replayed 1/1 log segments. Stats: ops{read=135 overwritten=0 applied=135 ignored=0} inserts{seen=804 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:11.926940 14178 tablet_bootstrap.cc:492] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478: Bootstrap complete.
I20240502 14:00:11.927157 14178 ts_tablet_manager.cc:1367] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478: Time spent bootstrapping tablet: real 0.230s user 0.110s sys 0.014s
I20240502 14:00:11.927997 14178 raft_consensus.cc:348] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } }
I20240502 14:00:11.928138 14178 raft_consensus.cc:729] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5fbb448689c949d98dec8e37cf5c3478, State: Initialized, Role: LEARNER
I20240502 14:00:11.928241 14178 consensus_queue.cc:260] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 135, Last appended: 1.135, Last appended by leader: 135, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: true } }
I20240502 14:00:11.928630 14178 ts_tablet_manager.cc:1398] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:11.929114 11788 tablet_copy_service.cc:342] P 73ce3d001cdf4d0c8e1487958b7694ea: Request end of tablet copy session 5fbb448689c949d98dec8e37cf5c3478-b9bc1c84cb144965a21bdc40f099416d received from {username='slave'} at 127.8.86.133:43393
I20240502 14:00:11.929198 11788 tablet_copy_service.cc:433] P 73ce3d001cdf4d0c8e1487958b7694ea: ending tablet copy session 5fbb448689c949d98dec8e37cf5c3478-b9bc1c84cb144965a21bdc40f099416d on tablet b9bc1c84cb144965a21bdc40f099416d with peer 5fbb448689c949d98dec8e37cf5c3478
I20240502 14:00:12.017683 13039 raft_consensus.cc:1203] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.153->[1.154-1.173] Dedup: 1.159->[1.160-1.173]
I20240502 14:00:12.035171 13431 raft_consensus.cc:1203] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6] Dedup: 2.6->[]
I20240502 14:00:12.049453 14150 tablet_bootstrap.cc:492] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766: Bootstrap replayed 1/1 log segments. Stats: ops{read=138 overwritten=0 applied=138 ignored=0} inserts{seen=810 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:12.049772 14150 tablet_bootstrap.cc:492] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766: Bootstrap complete.
I20240502 14:00:12.049966 14150 ts_tablet_manager.cc:1367] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766: Time spent bootstrapping tablet: real 0.418s user 0.128s sys 0.000s
I20240502 14:00:12.050426 14150 raft_consensus.cc:348] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } }
I20240502 14:00:12.050537 14150 raft_consensus.cc:729] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a36d5a88b1a34006bd3d040f5a715766, State: Initialized, Role: LEARNER
I20240502 14:00:12.050617 14150 consensus_queue.cc:260] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 138, Last appended: 1.138, Last appended by leader: 138, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } }
I20240502 14:00:12.050933 14150 ts_tablet_manager.cc:1398] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 14:00:12.062394 11919 tablet_copy_service.cc:342] P cbf92f56d7144a22a6d482c73c7c7ea9: Request end of tablet copy session a36d5a88b1a34006bd3d040f5a715766-182302167a0541888cb99bcd66635125 received from {username='slave'} at 127.8.86.134:45157
I20240502 14:00:12.062530 11919 tablet_copy_service.cc:433] P cbf92f56d7144a22a6d482c73c7c7ea9: ending tablet copy session a36d5a88b1a34006bd3d040f5a715766-182302167a0541888cb99bcd66635125 on tablet 182302167a0541888cb99bcd66635125 with peer a36d5a88b1a34006bd3d040f5a715766
I20240502 14:00:12.082248 13518 raft_consensus.cc:1050] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9: attempting to promote NON_VOTER 4f6f2d2b1569415cbbbea2466dc28e28 to VOTER
I20240502 14:00:12.082599 13518 consensus_queue.cc:237] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 174, Committed index: 174, Last appended: 1.175, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } }
I20240502 14:00:12.083844 13038 raft_consensus.cc:1261] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 LEARNER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 1 index: 175. Preceding OpId from leader: term: 1 index: 176. (index mismatch)
I20240502 14:00:12.084318 12808 consensus_queue.cc:1035] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 176, Last known committed idx: 174, Time since last communication: 0.000s
I20240502 14:00:12.087335 11899 raft_consensus.cc:1261] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 1 index: 175. Preceding OpId from leader: term: 1 index: 176. (index mismatch)
I20240502 14:00:12.088388 12808 raft_consensus.cc:2932] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Committing config change with OpId 1.176: config changed from index 154 to 176, 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) changed from NON_VOTER to VOTER. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:12.091918 11624 catalog_manager.cc:5484] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 reported cstate change: config changed from index 154 to 176, 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "65a078666d30488f9c204d93e6394ef9" committed_config { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:12.093900 13518 consensus_queue.cc:1035] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 176, Last known committed idx: 174, Time since last communication: 0.000s
I20240502 14:00:12.094365 13038 raft_consensus.cc:2932] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Committing config change with OpId 1.176: config changed from index 154 to 176, 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) changed from NON_VOTER to VOTER. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:12.097577 14173 tablet_bootstrap.cc:492] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766: Bootstrap replayed 1/1 log segments. Stats: ops{read=131 overwritten=0 applied=131 ignored=0} inserts{seen=762 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:12.097913 14173 tablet_bootstrap.cc:492] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766: Bootstrap complete.
I20240502 14:00:12.098121 14173 ts_tablet_manager.cc:1367] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766: Time spent bootstrapping tablet: real 0.451s user 0.118s sys 0.004s
I20240502 14:00:12.098627 14173 raft_consensus.cc:348] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } }
I20240502 14:00:12.098750 14173 raft_consensus.cc:729] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a36d5a88b1a34006bd3d040f5a715766, State: Initialized, Role: LEARNER
I20240502 14:00:12.098850 14173 consensus_queue.cc:260] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 131, Last appended: 1.131, Last appended by leader: 131, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: true } }
I20240502 14:00:12.099131 14173 ts_tablet_manager.cc:1398] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:12.105639 11899 raft_consensus.cc:2932] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Committing config change with OpId 1.176: config changed from index 154 to 176, 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132) changed from NON_VOTER to VOTER. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:12.109601 11919 tablet_copy_service.cc:342] P cbf92f56d7144a22a6d482c73c7c7ea9: Request end of tablet copy session a36d5a88b1a34006bd3d040f5a715766-986b926ead014774925e3b8a3892c56d received from {username='slave'} at 127.8.86.134:45157
I20240502 14:00:12.109694 11919 tablet_copy_service.cc:433] P cbf92f56d7144a22a6d482c73c7c7ea9: ending tablet copy session a36d5a88b1a34006bd3d040f5a715766-986b926ead014774925e3b8a3892c56d on tablet 986b926ead014774925e3b8a3892c56d with peer a36d5a88b1a34006bd3d040f5a715766
I20240502 14:00:12.238550 13172 raft_consensus.cc:1203] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.112->[1.113-1.152] Dedup: 1.135->[1.136-1.152]
I20240502 14:00:12.273458 13569 raft_consensus.cc:1050] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea: attempting to promote NON_VOTER 5fbb448689c949d98dec8e37cf5c3478 to VOTER
I20240502 14:00:12.273795 13569 consensus_queue.cc:237] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 153, Committed index: 153, Last appended: 1.153, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } }
I20240502 14:00:12.275444 13434 raft_consensus.cc:1261] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 153. Preceding OpId from leader: term: 1 index: 154. (index mismatch)
I20240502 14:00:12.276190 13559 consensus_queue.cc:1035] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 154, Last known committed idx: 153, Time since last communication: 0.000s
I20240502 14:00:12.277210 13172 raft_consensus.cc:1261] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 1 LEARNER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 153. Preceding OpId from leader: term: 1 index: 154. (index mismatch)
I20240502 14:00:12.277668 13559 consensus_queue.cc:1035] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 154, Last known committed idx: 153, Time since last communication: 0.000s
I20240502 14:00:12.279750 13559 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Committing config change with OpId 1.154: config changed from index 113 to 154, 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
I20240502 14:00:12.282023 13170 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 1 FOLLOWER]: Committing config change with OpId 1.154: config changed from index 113 to 154, 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
I20240502 14:00:12.283105 11624 catalog_manager.cc:5484] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: config changed from index 113 to 154, 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:12.287451 13434 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 1 FOLLOWER]: Committing config change with OpId 1.154: config changed from index 113 to 154, 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
I20240502 14:00:12.373179 12030 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db4ce5e277504649a781e23861baa667"
dest_uuid: "65a078666d30488f9c204d93e6394ef9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49050
I20240502 14:00:12.373355 12030 raft_consensus.cc:595] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Received request to transfer leadership
W20240502 14:00:12.392881 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
W20240502 14:00:12.399174 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
W20240502 14:00:12.402427 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
W20240502 14:00:12.408649 11989 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
W20240502 14:00:12.417451 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
W20240502 14:00:12.427006 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
W20240502 14:00:12.435174 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
I20240502 14:00:12.436565 11767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b9bc1c84cb144965a21bdc40f099416d"
dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53524
I20240502 14:00:12.436693 11767 raft_consensus.cc:595] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:12.438860 13559 raft_consensus.cc:979] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea: : Instructing follower 7553d1bab6f642ac8395c928da1c58d7 to start an election
I20240502 14:00:12.438961 13559 raft_consensus.cc:1067] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Signalling peer 7553d1bab6f642ac8395c928da1c58d7 to start an election
I20240502 14:00:12.440090 13433 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9bc1c84cb144965a21bdc40f099416d"
dest_uuid: "7553d1bab6f642ac8395c928da1c58d7"
from {username='slave'} at 127.8.86.129:37195
I20240502 14:00:12.440284 13433 raft_consensus.cc:482] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:12.440403 13433 raft_consensus.cc:3037] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:00:12.441020 11727 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53214: Service unavailable: leader transfer in progress
I20240502 14:00:12.441301 13433 raft_consensus.cc:504] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } }
I20240502 14:00:12.442230 13433 leader_election.cc:288] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [CANDIDATE]: Term 2 election: Requested vote from peers 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411), 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133:44919)
I20240502 14:00:12.443069 11767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9bc1c84cb144965a21bdc40f099416d" candidate_uuid: "7553d1bab6f642ac8395c928da1c58d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 158 } } ignore_live_leader: true dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
I20240502 14:00:12.443183 11767 raft_consensus.cc:3032] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:12.443215 11767 raft_consensus.cc:729] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Running, Role: LEADER
W20240502 14:00:12.443270 11727 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53214: Service unavailable: leader transfer in progress
I20240502 14:00:12.443336 11767 consensus_queue.cc:260] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 158, Committed index: 158, Last appended: 1.158, Last appended by leader: 158, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } }
I20240502 14:00:12.443462 11767 raft_consensus.cc:3037] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:12.444480 11767 raft_consensus.cc:2454] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7553d1bab6f642ac8395c928da1c58d7 in term 2.
W20240502 14:00:12.444715 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
I20240502 14:00:12.447917 13369 leader_election.cc:302] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73ce3d001cdf4d0c8e1487958b7694ea, 7553d1bab6f642ac8395c928da1c58d7; no voters:
I20240502 14:00:12.448453 14166 raft_consensus.cc:2781] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:12.448639 14166 raft_consensus.cc:686] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 2 LEADER]: Becoming Leader. State: Replica: 7553d1bab6f642ac8395c928da1c58d7, State: Running, Role: LEADER
I20240502 14:00:12.448974 14166 consensus_queue.cc:237] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 158, Committed index: 158, Last appended: 1.158, Last appended by leader: 158, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } }
W20240502 14:00:12.449331 11727 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53214: Illegal state: replica 73ce3d001cdf4d0c8e1487958b7694ea is not leader of this config: current role FOLLOWER
I20240502 14:00:12.451857 11620 catalog_manager.cc:5484] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 reported cstate change: term changed from 1 to 2, leader changed from 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) to 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135). New cstate: current_term: 2 leader_uuid: "7553d1bab6f642ac8395c928da1c58d7" committed_config { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:12.455148 13170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9bc1c84cb144965a21bdc40f099416d" candidate_uuid: "7553d1bab6f642ac8395c928da1c58d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 158 } } ignore_live_leader: true dest_uuid: "5fbb448689c949d98dec8e37cf5c3478"
I20240502 14:00:12.455276 13170 raft_consensus.cc:3037] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:12.456451 13170 raft_consensus.cc:2454] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7553d1bab6f642ac8395c928da1c58d7 in term 2.
W20240502 14:00:12.460062 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
W20240502 14:00:12.471694 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
W20240502 14:00:12.487198 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
I20240502 14:00:12.494272 13303 raft_consensus.cc:1203] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.119->[1.120-1.168] Dedup: 1.131->[1.132-1.168]
W20240502 14:00:12.502553 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
I20240502 14:00:12.510932 11765 raft_consensus.cc:1261] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Refusing update from remote peer 7553d1bab6f642ac8395c928da1c58d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 158. Preceding OpId from leader: term: 2 index: 160. (index mismatch)
I20240502 14:00:12.514717 13172 raft_consensus.cc:1261] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Refusing update from remote peer 7553d1bab6f642ac8395c928da1c58d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 158. Preceding OpId from leader: term: 2 index: 160. (index mismatch)
I20240502 14:00:12.516451 14166 consensus_queue.cc:1035] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 158, Time since last communication: 0.000s
I20240502 14:00:12.516932 14166 consensus_queue.cc:1035] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 158, Time since last communication: 0.000s
W20240502 14:00:12.518689 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
I20240502 14:00:12.528265 14224 mvcc.cc:205] Tried to move back new op lower bound from 7023240857618296832 to 7023240857394241536. Current Snapshot: MvccSnapshot[applied={T|T < 7023240857618296832}]
W20240502 14:00:12.538480 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
I20240502 14:00:12.538837 13301 raft_consensus.cc:1203] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.112->[1.113-1.168] Dedup: 1.138->[1.139-1.168]
W20240502 14:00:12.554127 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
I20240502 14:00:12.559656 13434 consensus_queue.cc:237] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 160, Committed index: 160, Last appended: 2.161, Last appended by leader: 158, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 162 OBSOLETE_local: false peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } }
I20240502 14:00:12.561450 13170 raft_consensus.cc:1261] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Refusing update from remote peer 7553d1bab6f642ac8395c928da1c58d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 161. Preceding OpId from leader: term: 2 index: 162. (index mismatch)
I20240502 14:00:12.563186 14166 consensus_queue.cc:1035] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 162, Last known committed idx: 160, Time since last communication: 0.000s
I20240502 14:00:12.565623 14263 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 2 LEADER]: Committing config change with OpId 2.162: config changed from index 154 to 162, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New config: { opid_index: 162 OBSOLETE_local: false peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
I20240502 14:00:12.568539 11620 catalog_manager.cc:5484] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 reported cstate change: config changed from index 154 to 162, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New cstate: current_term: 2 leader_uuid: "7553d1bab6f642ac8395c928da1c58d7" committed_config { opid_index: 162 OBSOLETE_local: false peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:12.573726 11613 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b9bc1c84cb144965a21bdc40f099416d with cas_config_opid_index 154: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:12.574090 11747 tablet_service.cc:1514] Processing DeleteTablet for tablet b9bc1c84cb144965a21bdc40f099416d with delete_type TABLET_DATA_TOMBSTONED (TS 73ce3d001cdf4d0c8e1487958b7694ea not found in new config with opid_index 162) from {username='slave'} at 127.0.0.1:43190
I20240502 14:00:12.574002 13170 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Committing config change with OpId 2.162: config changed from index 154 to 162, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New config: { opid_index: 162 OBSOLETE_local: false peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } }
W20240502 14:00:12.574687 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
I20240502 14:00:12.576100 13661 raft_consensus.cc:1050] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9: attempting to promote NON_VOTER a36d5a88b1a34006bd3d040f5a715766 to VOTER
I20240502 14:00:12.576527 13661 consensus_queue.cc:237] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 169, Committed index: 169, Last appended: 1.169, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } }
I20240502 14:00:12.579228 14280 tablet_replica.cc:330] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea: stopping tablet replica
I20240502 14:00:12.580163 13301 raft_consensus.cc:1261] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 [term 1 LEARNER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 169. Preceding OpId from leader: term: 1 index: 170. (index mismatch)
I20240502 14:00:12.580984 14280 raft_consensus.cc:2229] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:12.581197 14280 raft_consensus.cc:2258] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:12.581529 14123 consensus_queue.cc:1035] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 170, Last known committed idx: 169, Time since last communication: 0.000s
I20240502 14:00:12.582111 14280 ts_tablet_manager.cc:1875] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:12.585188 11765 raft_consensus.cc:1261] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 169. Preceding OpId from leader: term: 1 index: 170. (index mismatch)
I20240502 14:00:12.586383 14123 raft_consensus.cc:1050] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9: attempting to promote NON_VOTER a36d5a88b1a34006bd3d040f5a715766 to VOTER
I20240502 14:00:12.586606 14280 ts_tablet_manager.cc:1888] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.160
I20240502 14:00:12.586747 14280 log.cc:1196] T b9bc1c84cb144965a21bdc40f099416d P 73ce3d001cdf4d0c8e1487958b7694ea: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/wals/b9bc1c84cb144965a21bdc40f099416d
I20240502 14:00:12.586699 14123 consensus_queue.cc:237] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 170, Committed index: 170, Last appended: 1.171, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } }
I20240502 14:00:12.588114 13539 raft_consensus.cc:1013] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: attempt to promote peer a36d5a88b1a34006bd3d040f5a715766: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:00:12.588110 11612 catalog_manager.cc:4830] TS 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411): tablet b9bc1c84cb144965a21bdc40f099416d (table rebalance_test_table_4 [id=83b2409333254a509079ef88e0c156f9]) successfully deleted
I20240502 14:00:12.589413 13539 raft_consensus.cc:2932] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Committing config change with OpId 1.170: config changed from index 113 to 170, a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:12.589836 13303 raft_consensus.cc:1261] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 [term 1 LEARNER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 171. Preceding OpId from leader: term: 1 index: 172. (index mismatch)
I20240502 14:00:12.591758 14123 consensus_queue.cc:1035] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 172, Last known committed idx: 170, Time since last communication: 0.000s
I20240502 14:00:12.591778 13303 raft_consensus.cc:2932] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 [term 1 FOLLOWER]: Committing config change with OpId 1.170: config changed from index 113 to 170, a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:12.592952 11620 catalog_manager.cc:5484] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: config changed from index 113 to 170, a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:00:12.594676 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
I20240502 14:00:12.594858 13783 consensus_queue.cc:1035] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 170, Last known committed idx: 169, Time since last communication: 0.000s
I20240502 14:00:12.597077 12027 raft_consensus.cc:1261] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 171. Preceding OpId from leader: term: 1 index: 172. (index mismatch)
I20240502 14:00:12.597491 11765 raft_consensus.cc:2932] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Committing config change with OpId 1.170: config changed from index 113 to 170, a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:12.598567 13572 raft_consensus.cc:2932] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Committing config change with OpId 1.172: config changed from index 120 to 172, a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:12.601366 13303 raft_consensus.cc:2932] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 [term 1 FOLLOWER]: Committing config change with OpId 1.172: config changed from index 120 to 172, a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:12.604116 11624 catalog_manager.cc:5484] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: config changed from index 120 to 172, a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:12.605700 13572 consensus_queue.cc:1035] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 172, Last known committed idx: 170, Time since last communication: 0.000s
I20240502 14:00:12.612975 12026 raft_consensus.cc:2932] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Committing config change with OpId 1.172: config changed from index 120 to 172, a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:12.615886 13506 raft_consensus.cc:1050] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea: attempting to promote NON_VOTER 7553d1bab6f642ac8395c928da1c58d7 to VOTER
I20240502 14:00:12.616261 13506 consensus_queue.cc:237] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } }
W20240502 14:00:12.616560 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
I20240502 14:00:12.617815 13433 raft_consensus.cc:1261] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7 [term 2 LEARNER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:00:12.618533 13170 raft_consensus.cc:1261] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:00:12.618795 13569 consensus_queue.cc:1035] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:12.619196 13569 consensus_queue.cc:1035] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:12.621062 13699 raft_consensus.cc:2932] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:12.623881 13170 raft_consensus.cc:2932] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:12.627857 11624 catalog_manager.cc:5484] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 reported cstate change: config changed from index 6 to 7, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
W20240502 14:00:12.640345 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
W20240502 14:00:12.662663 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
I20240502 14:00:12.672998 13433 raft_consensus.cc:2932] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
W20240502 14:00:12.688473 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
W20240502 14:00:12.713929 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
I20240502 14:00:12.734143 11899 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "986b926ead014774925e3b8a3892c56d"
dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38818
I20240502 14:00:12.736750 11899 raft_consensus.cc:595] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Received request to transfer leadership
W20240502 14:00:12.742434 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
I20240502 14:00:12.743541 13661 raft_consensus.cc:979] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9: : Instructing follower a36d5a88b1a34006bd3d040f5a715766 to start an election
I20240502 14:00:12.743643 13661 raft_consensus.cc:1067] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Signalling peer a36d5a88b1a34006bd3d040f5a715766 to start an election
I20240502 14:00:12.748937 13302 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "986b926ead014774925e3b8a3892c56d"
dest_uuid: "a36d5a88b1a34006bd3d040f5a715766"
from {username='slave'} at 127.8.86.130:56983
I20240502 14:00:12.749115 13302 raft_consensus.cc:482] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:12.749207 13302 raft_consensus.cc:3037] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:12.750231 13302 raft_consensus.cc:504] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } }
I20240502 14:00:12.751278 13302 leader_election.cc:288] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 [CANDIDATE]: Term 2 election: Requested vote from peers cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377), 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:12.755997 11899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986b926ead014774925e3b8a3892c56d" candidate_uuid: "a36d5a88b1a34006bd3d040f5a715766" candidate_term: 2 candidate_status { last_received { term: 1 index: 181 } } ignore_live_leader: true dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9"
I20240502 14:00:12.756116 11899 raft_consensus.cc:3032] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:12.756158 11899 raft_consensus.cc:729] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Running, Role: LEADER
I20240502 14:00:12.756289 11899 consensus_queue.cc:260] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 181, Committed index: 181, Last appended: 1.181, Last appended by leader: 181, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } }
I20240502 14:00:12.756467 11899 raft_consensus.cc:3037] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:12.757508 11899 raft_consensus.cc:2454] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36d5a88b1a34006bd3d040f5a715766 in term 2.
I20240502 14:00:12.760950 13239 leader_election.cc:302] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a36d5a88b1a34006bd3d040f5a715766, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:12.761215 14220 raft_consensus.cc:2781] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:12.761411 14220 raft_consensus.cc:686] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 [term 2 LEADER]: Becoming Leader. State: Replica: a36d5a88b1a34006bd3d040f5a715766, State: Running, Role: LEADER
I20240502 14:00:12.761741 14220 consensus_queue.cc:237] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 181, Committed index: 181, Last appended: 1.181, Last appended by leader: 181, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } }
I20240502 14:00:12.764037 11620 catalog_manager.cc:5484] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 reported cstate change: term changed from 1 to 2, leader changed from cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) to a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134). New cstate: current_term: 2 leader_uuid: "a36d5a88b1a34006bd3d040f5a715766" committed_config { opid_index: 172 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:00:12.769642 11989 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
W20240502 14:00:12.781721 11857 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38560: Illegal state: replica cbf92f56d7144a22a6d482c73c7c7ea9 is not leader of this config: current role FOLLOWER
I20240502 14:00:12.793395 13686 raft_consensus.cc:979] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9: : Instructing follower cbf92f56d7144a22a6d482c73c7c7ea9 to start an election
I20240502 14:00:12.793504 13686 raft_consensus.cc:1067] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Signalling peer cbf92f56d7144a22a6d482c73c7c7ea9 to start an election
I20240502 14:00:12.795120 11899 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db4ce5e277504649a781e23861baa667"
dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9"
from {username='slave'} at 127.8.86.131:54755
I20240502 14:00:12.795256 11899 raft_consensus.cc:482] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:12.795311 11899 raft_consensus.cc:3037] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:00:12.795809 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Service unavailable: leader transfer in progress
I20240502 14:00:12.796475 11899 raft_consensus.cc:504] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } }
I20240502 14:00:12.796931 11899 leader_election.cc:288] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 2 election: Requested vote from peers 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325), 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132:42709)
I20240502 14:00:12.803118 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ce5e277504649a781e23861baa667" candidate_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" candidate_term: 2 candidate_status { last_received { term: 1 index: 187 } } ignore_live_leader: true dest_uuid: "4f6f2d2b1569415cbbbea2466dc28e28"
I20240502 14:00:12.803306 13038 raft_consensus.cc:3037] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:12.803922 12027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4ce5e277504649a781e23861baa667" candidate_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" candidate_term: 2 candidate_status { last_received { term: 1 index: 187 } } ignore_live_leader: true dest_uuid: "65a078666d30488f9c204d93e6394ef9"
I20240502 14:00:12.804034 12027 raft_consensus.cc:3032] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:12.804075 12027 raft_consensus.cc:729] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 65a078666d30488f9c204d93e6394ef9, State: Running, Role: LEADER
I20240502 14:00:12.804224 12027 consensus_queue.cc:260] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 187, Committed index: 187, Last appended: 1.187, Last appended by leader: 187, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } }
I20240502 14:00:12.804399 12027 raft_consensus.cc:3037] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:12.804487 13038 raft_consensus.cc:2454] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbf92f56d7144a22a6d482c73c7c7ea9 in term 2.
I20240502 14:00:12.805444 12027 raft_consensus.cc:2454] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbf92f56d7144a22a6d482c73c7c7ea9 in term 2.
I20240502 14:00:12.805691 11835 leader_election.cc:302] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f6f2d2b1569415cbbbea2466dc28e28, cbf92f56d7144a22a6d482c73c7c7ea9; no voters:
I20240502 14:00:12.805859 13783 raft_consensus.cc:2781] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:12.805946 13783 raft_consensus.cc:686] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 LEADER]: Becoming Leader. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Running, Role: LEADER
I20240502 14:00:12.806068 13783 consensus_queue.cc:237] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 187, Committed index: 187, Last appended: 1.187, Last appended by leader: 187, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } }
I20240502 14:00:12.809047 11620 catalog_manager.cc:5484] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: term changed from 1 to 2, leader changed from 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) to cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130). New cstate: current_term: 2 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 14:00:12.809331 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49070: Illegal state: replica 65a078666d30488f9c204d93e6394ef9 is not leader of this config: current role FOLLOWER
I20240502 14:00:12.812290 12027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986b926ead014774925e3b8a3892c56d" candidate_uuid: "a36d5a88b1a34006bd3d040f5a715766" candidate_term: 2 candidate_status { last_received { term: 1 index: 181 } } ignore_live_leader: true dest_uuid: "65a078666d30488f9c204d93e6394ef9"
I20240502 14:00:12.812399 12027 raft_consensus.cc:3037] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:12.814440 12027 raft_consensus.cc:2454] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36d5a88b1a34006bd3d040f5a715766 in term 2.
W20240502 14:00:12.827350 11990 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48868: Illegal state: replica 65a078666d30488f9c204d93e6394ef9 is not leader of this config: current role FOLLOWER
I20240502 14:00:12.828472 11765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "03409b5f413f499c919f2c6a5a4126f7"
dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53538
I20240502 14:00:12.828605 11765 raft_consensus.cc:595] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 LEADER]: Received request to transfer leadership
I20240502 14:00:12.837466 12027 raft_consensus.cc:1261] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9 [term 2 FOLLOWER]: Refusing update from remote peer a36d5a88b1a34006bd3d040f5a715766: Log matching property violated. Preceding OpId in replica: term: 1 index: 181. Preceding OpId from leader: term: 2 index: 183. (index mismatch)
I20240502 14:00:12.838068 11899 raft_consensus.cc:1261] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Refusing update from remote peer a36d5a88b1a34006bd3d040f5a715766: Log matching property violated. Preceding OpId in replica: term: 1 index: 181. Preceding OpId from leader: term: 2 index: 183. (index mismatch)
I20240502 14:00:12.838622 14220 consensus_queue.cc:1035] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I20240502 14:00:12.840555 14220 consensus_queue.cc:1035] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I20240502 14:00:12.844039 14278 mvcc.cc:205] Tried to move back new op lower bound from 7023240858976899072 to 7023240858674974720. Current Snapshot: MvccSnapshot[applied={T|T < 7023240858976899072 or (T in {7023240858976899072})}]
I20240502 14:00:12.879973 13303 consensus_queue.cc:237] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 184, Committed index: 184, Last appended: 2.184, Last appended by leader: 181, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 185 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } }
I20240502 14:00:12.881425 12026 raft_consensus.cc:1261] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9 [term 2 FOLLOWER]: Refusing update from remote peer a36d5a88b1a34006bd3d040f5a715766: Log matching property violated. Preceding OpId in replica: term: 2 index: 184. Preceding OpId from leader: term: 2 index: 185. (index mismatch)
I20240502 14:00:12.882120 14299 consensus_queue.cc:1035] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 185, Last known committed idx: 184, Time since last communication: 0.000s
I20240502 14:00:12.884680 14299 raft_consensus.cc:2932] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 [term 2 LEADER]: Committing config change with OpId 2.185: config changed from index 172 to 185, VOTER cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) evicted. New config: { opid_index: 185 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:12.886569 11610 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 986b926ead014774925e3b8a3892c56d with cas_config_opid_index 172: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:12.887028 12026 raft_consensus.cc:2932] T 986b926ead014774925e3b8a3892c56d P 65a078666d30488f9c204d93e6394ef9 [term 2 FOLLOWER]: Committing config change with OpId 2.185: config changed from index 172 to 185, VOTER cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) evicted. New config: { opid_index: 185 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:12.887517 11624 catalog_manager.cc:5484] T 986b926ead014774925e3b8a3892c56d P a36d5a88b1a34006bd3d040f5a715766 reported cstate change: config changed from index 172 to 185, VOTER cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) evicted. New cstate: current_term: 2 leader_uuid: "a36d5a88b1a34006bd3d040f5a715766" committed_config { opid_index: 185 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:12.897989 11877 tablet_service.cc:1514] Processing DeleteTablet for tablet 986b926ead014774925e3b8a3892c56d with delete_type TABLET_DATA_TOMBSTONED (TS cbf92f56d7144a22a6d482c73c7c7ea9 not found in new config with opid_index 185) from {username='slave'} at 127.0.0.1:37684
I20240502 14:00:12.898794 14325 tablet_replica.cc:330] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9: stopping tablet replica
I20240502 14:00:12.898926 14325 raft_consensus.cc:2229] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:12.899061 14325 raft_consensus.cc:2258] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:12.900772 14325 ts_tablet_manager.cc:1875] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:12.904959 14325 ts_tablet_manager.cc:1888] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.184
I20240502 14:00:12.905052 14325 log.cc:1196] T 986b926ead014774925e3b8a3892c56d P cbf92f56d7144a22a6d482c73c7c7ea9: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/wals/986b926ead014774925e3b8a3892c56d
I20240502 14:00:12.905855 13039 raft_consensus.cc:1261] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 2 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 187. Preceding OpId from leader: term: 2 index: 189. (index mismatch)
I20240502 14:00:12.906338 13661 consensus_queue.cc:1035] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 188, Last known committed idx: 187, Time since last communication: 0.000s
I20240502 14:00:12.906526 11613 catalog_manager.cc:4830] TS cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377): tablet 986b926ead014774925e3b8a3892c56d (table rebalance_test_table_1 [id=d6a2f7f1e4dc45ffbe7f9f2fba15577e]) successfully deleted
I20240502 14:00:12.906757 12027 raft_consensus.cc:1261] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 2 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 1 index: 187. Preceding OpId from leader: term: 2 index: 189. (index mismatch)
I20240502 14:00:12.908543 13661 consensus_queue.cc:1035] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 188, Last known committed idx: 187, Time since last communication: 0.000s
I20240502 14:00:12.911739 13835 mvcc.cc:205] Tried to move back new op lower bound from 7023240859257954304 to 7023240858855153664. Current Snapshot: MvccSnapshot[applied={T|T < 7023240859257954304}]
I20240502 14:00:12.914455 11899 consensus_queue.cc:237] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 189, Committed index: 189, Last appended: 2.189, Last appended by leader: 187, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 190 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } }
I20240502 14:00:12.918774 13040 raft_consensus.cc:1261] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 2 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 2 index: 189. Preceding OpId from leader: term: 2 index: 190. (index mismatch)
I20240502 14:00:12.922039 13783 consensus_queue.cc:1035] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 190, Last known committed idx: 189, Time since last communication: 0.000s
I20240502 14:00:12.926409 13539 raft_consensus.cc:2932] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 LEADER]: Committing config change with OpId 2.190: config changed from index 176 to 190, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New config: { opid_index: 190 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:12.928068 13038 raft_consensus.cc:2932] T db4ce5e277504649a781e23861baa667 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 2 FOLLOWER]: Committing config change with OpId 2.190: config changed from index 176 to 190, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New config: { opid_index: 190 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } }
I20240502 14:00:12.928788 11613 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet db4ce5e277504649a781e23861baa667 with cas_config_opid_index 176: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:12.929200 11624 catalog_manager.cc:5484] T db4ce5e277504649a781e23861baa667 P cbf92f56d7144a22a6d482c73c7c7ea9 reported cstate change: config changed from index 176 to 190, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New cstate: current_term: 2 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: 190 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:12.933257 12008 tablet_service.cc:1514] Processing DeleteTablet for tablet db4ce5e277504649a781e23861baa667 with delete_type TABLET_DATA_TOMBSTONED (TS 65a078666d30488f9c204d93e6394ef9 not found in new config with opid_index 190) from {username='slave'} at 127.0.0.1:38592
I20240502 14:00:12.934693 14328 tablet_replica.cc:330] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9: stopping tablet replica
I20240502 14:00:12.934896 14328 raft_consensus.cc:2229] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:12.936012 14328 raft_consensus.cc:2258] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:12.939138 14328 ts_tablet_manager.cc:1875] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:12.945328 14328 ts_tablet_manager.cc:1888] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.189
I20240502 14:00:12.945485 14328 log.cc:1196] T db4ce5e277504649a781e23861baa667 P 65a078666d30488f9c204d93e6394ef9: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/wals/db4ce5e277504649a781e23861baa667
I20240502 14:00:12.948792 11612 catalog_manager.cc:4830] TS 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325): tablet db4ce5e277504649a781e23861baa667 (table rebalance_test_table_0 [id=610fb42d93954513952217d0731890d9]) successfully deleted
I20240502 14:00:13.031875 11899 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "182302167a0541888cb99bcd66635125"
dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38848
I20240502 14:00:13.032034 11899 raft_consensus.cc:595] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:13.038151 13783 raft_consensus.cc:979] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9: : Instructing follower a36d5a88b1a34006bd3d040f5a715766 to start an election
I20240502 14:00:13.038453 13783 raft_consensus.cc:1067] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Signalling peer a36d5a88b1a34006bd3d040f5a715766 to start an election
I20240502 14:00:13.042320 13303 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "182302167a0541888cb99bcd66635125"
dest_uuid: "a36d5a88b1a34006bd3d040f5a715766"
from {username='slave'} at 127.8.86.130:56983
I20240502 14:00:13.042465 13303 raft_consensus.cc:482] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:13.042533 13303 raft_consensus.cc:3037] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:13.043686 13303 raft_consensus.cc:504] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } }
I20240502 14:00:13.045734 13303 leader_election.cc:288] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 [CANDIDATE]: Term 2 election: Requested vote from peers 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411), cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377)
I20240502 14:00:13.046344 11765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182302167a0541888cb99bcd66635125" candidate_uuid: "a36d5a88b1a34006bd3d040f5a715766" candidate_term: 2 candidate_status { last_received { term: 1 index: 188 } } ignore_live_leader: true dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
I20240502 14:00:13.046480 11765 raft_consensus.cc:3037] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:13.047626 11765 raft_consensus.cc:2454] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36d5a88b1a34006bd3d040f5a715766 in term 2.
I20240502 14:00:13.048940 13238 leader_election.cc:302] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73ce3d001cdf4d0c8e1487958b7694ea, a36d5a88b1a34006bd3d040f5a715766; no voters:
I20240502 14:00:13.049124 14299 raft_consensus.cc:2781] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:13.049189 14299 raft_consensus.cc:686] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 [term 2 LEADER]: Becoming Leader. State: Replica: a36d5a88b1a34006bd3d040f5a715766, State: Running, Role: LEADER
I20240502 14:00:13.049316 14299 consensus_queue.cc:237] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 188, Committed index: 188, Last appended: 1.188, Last appended by leader: 188, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } }
I20240502 14:00:13.050310 11899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "182302167a0541888cb99bcd66635125" candidate_uuid: "a36d5a88b1a34006bd3d040f5a715766" candidate_term: 2 candidate_status { last_received { term: 1 index: 188 } } ignore_live_leader: true dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9"
I20240502 14:00:13.050428 11899 raft_consensus.cc:3032] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:13.050468 11899 raft_consensus.cc:729] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Running, Role: LEADER
I20240502 14:00:13.050611 11899 consensus_queue.cc:260] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 188, Committed index: 188, Last appended: 1.188, Last appended by leader: 188, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } }
I20240502 14:00:13.050781 11899 raft_consensus.cc:3037] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:13.051844 11899 raft_consensus.cc:2454] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36d5a88b1a34006bd3d040f5a715766 in term 2.
I20240502 14:00:13.052321 11620 catalog_manager.cc:5484] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 reported cstate change: term changed from 1 to 2, leader changed from cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) to a36d5a88b1a34006bd3d040f5a715766 (127.8.86.134). New cstate: current_term: 2 leader_uuid: "a36d5a88b1a34006bd3d040f5a715766" committed_config { opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:00:13.091154 11856 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38568: Illegal state: replica cbf92f56d7144a22a6d482c73c7c7ea9 is not leader of this config: current role FOLLOWER
W20240502 14:00:13.109411 11728 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53544: Illegal state: replica 73ce3d001cdf4d0c8e1487958b7694ea is not leader of this config: current role FOLLOWER
I20240502 14:00:13.124809 11899 raft_consensus.cc:1261] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Refusing update from remote peer a36d5a88b1a34006bd3d040f5a715766: Log matching property violated. Preceding OpId in replica: term: 1 index: 188. Preceding OpId from leader: term: 2 index: 190. (index mismatch)
I20240502 14:00:13.125517 14220 consensus_queue.cc:1035] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 189, Last known committed idx: 188, Time since last communication: 0.000s
I20240502 14:00:13.125676 11765 raft_consensus.cc:1261] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Refusing update from remote peer a36d5a88b1a34006bd3d040f5a715766: Log matching property violated. Preceding OpId in replica: term: 1 index: 188. Preceding OpId from leader: term: 2 index: 190. (index mismatch)
I20240502 14:00:13.127419 14220 consensus_queue.cc:1035] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 189, Last known committed idx: 188, Time since last communication: 0.000s
I20240502 14:00:13.135776 13303 consensus_queue.cc:237] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 190, Committed index: 190, Last appended: 2.190, Last appended by leader: 188, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 191 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } }
I20240502 14:00:13.139436 11765 raft_consensus.cc:1261] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Refusing update from remote peer a36d5a88b1a34006bd3d040f5a715766: Log matching property violated. Preceding OpId in replica: term: 2 index: 190. Preceding OpId from leader: term: 2 index: 191. (index mismatch)
I20240502 14:00:13.141245 14299 consensus_queue.cc:1035] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 191, Last known committed idx: 190, Time since last communication: 0.000s
I20240502 14:00:13.145059 13559 raft_consensus.cc:979] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea: : Instructing follower 5fbb448689c949d98dec8e37cf5c3478 to start an election
I20240502 14:00:13.145161 13559 raft_consensus.cc:1067] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 LEADER]: Signalling peer 5fbb448689c949d98dec8e37cf5c3478 to start an election
I20240502 14:00:13.145687 13172 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "03409b5f413f499c919f2c6a5a4126f7"
dest_uuid: "5fbb448689c949d98dec8e37cf5c3478"
from {username='slave'} at 127.8.86.129:42107
I20240502 14:00:13.145860 13172 raft_consensus.cc:482] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:13.145912 13172 raft_consensus.cc:3037] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:00:13.146975 13172 raft_consensus.cc:504] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } }
I20240502 14:00:13.148355 14220 raft_consensus.cc:2932] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 [term 2 LEADER]: Committing config change with OpId 2.191: config changed from index 170 to 191, VOTER cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) evicted. New config: { opid_index: 191 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:13.150167 11610 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 182302167a0541888cb99bcd66635125 with cas_config_opid_index 170: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:13.155359 11765 raft_consensus.cc:2932] T 182302167a0541888cb99bcd66635125 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Committing config change with OpId 2.191: config changed from index 170 to 191, VOTER cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) evicted. New config: { opid_index: 191 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } } }
I20240502 14:00:13.156831 11624 catalog_manager.cc:5484] T 182302167a0541888cb99bcd66635125 P a36d5a88b1a34006bd3d040f5a715766 reported cstate change: config changed from index 170 to 191, VOTER cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) evicted. New cstate: current_term: 2 leader_uuid: "a36d5a88b1a34006bd3d040f5a715766" committed_config { opid_index: 191 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a36d5a88b1a34006bd3d040f5a715766" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 36541 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:13.158782 11767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "03409b5f413f499c919f2c6a5a4126f7"
dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53560
I20240502 14:00:13.158900 11767 raft_consensus.cc:595] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 LEADER]: Received request to transfer leadership
I20240502 14:00:13.159113 11768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03409b5f413f499c919f2c6a5a4126f7" candidate_uuid: "5fbb448689c949d98dec8e37cf5c3478" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
I20240502 14:00:13.159163 13172 leader_election.cc:288] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [CANDIDATE]: Term 3 election: Requested vote from peers 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411), 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135:34019)
I20240502 14:00:13.159226 11768 raft_consensus.cc:3032] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:00:13.159262 11768 raft_consensus.cc:729] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Running, Role: LEADER
I20240502 14:00:13.159394 11768 consensus_queue.cc:260] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } }
I20240502 14:00:13.159555 11768 raft_consensus.cc:3037] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:00:13.159912 11877 tablet_service.cc:1514] Processing DeleteTablet for tablet 182302167a0541888cb99bcd66635125 with delete_type TABLET_DATA_TOMBSTONED (TS cbf92f56d7144a22a6d482c73c7c7ea9 not found in new config with opid_index 191) from {username='slave'} at 127.0.0.1:37684
I20240502 14:00:13.160132 14325 tablet_replica.cc:330] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9: stopping tablet replica
I20240502 14:00:13.160259 14325 raft_consensus.cc:2229] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:13.160372 14325 raft_consensus.cc:2258] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:13.160555 11768 raft_consensus.cc:2454] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fbb448689c949d98dec8e37cf5c3478 in term 3.
I20240502 14:00:13.161000 13106 leader_election.cc:302] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fbb448689c949d98dec8e37cf5c3478, 73ce3d001cdf4d0c8e1487958b7694ea; no voters:
I20240502 14:00:13.161480 14325 ts_tablet_manager.cc:1875] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:13.161877 14364 raft_consensus.cc:2781] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:00:13.162022 14364 raft_consensus.cc:686] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 3 LEADER]: Becoming Leader. State: Replica: 5fbb448689c949d98dec8e37cf5c3478, State: Running, Role: LEADER
I20240502 14:00:13.162340 14364 consensus_queue.cc:237] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } }
I20240502 14:00:13.163686 14325 ts_tablet_manager.cc:1888] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.190
I20240502 14:00:13.163758 14325 log.cc:1196] T 182302167a0541888cb99bcd66635125 P cbf92f56d7144a22a6d482c73c7c7ea9: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/wals/182302167a0541888cb99bcd66635125
I20240502 14:00:13.165830 11613 catalog_manager.cc:4830] TS cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377): tablet 182302167a0541888cb99bcd66635125 (table rebalance_test_table_3 [id=22735e7421b84776ac7e9ae79a0aa1f2]) successfully deleted
I20240502 14:00:13.166635 11617 catalog_manager.cc:5484] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 reported cstate change: term changed from 2 to 3, leader changed from 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) to 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133). New cstate: current_term: 3 leader_uuid: "5fbb448689c949d98dec8e37cf5c3478" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:13.201738 13433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03409b5f413f499c919f2c6a5a4126f7" candidate_uuid: "5fbb448689c949d98dec8e37cf5c3478" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "7553d1bab6f642ac8395c928da1c58d7"
I20240502 14:00:13.201936 13433 raft_consensus.cc:3037] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:00:13.203135 13433 raft_consensus.cc:2454] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fbb448689c949d98dec8e37cf5c3478 in term 3.
I20240502 14:00:13.611003 11765 raft_consensus.cc:1261] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 3 FOLLOWER]: Refusing update from remote peer 5fbb448689c949d98dec8e37cf5c3478: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240502 14:00:13.611629 14365 consensus_queue.cc:1035] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:13.616966 13434 raft_consensus.cc:1261] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7 [term 3 FOLLOWER]: Refusing update from remote peer 5fbb448689c949d98dec8e37cf5c3478: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240502 14:00:13.617478 14364 consensus_queue.cc:1035] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:14.154997 13169 consensus_queue.cc:237] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } }
I20240502 14:00:14.156106 13433 raft_consensus.cc:1261] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7 [term 3 FOLLOWER]: Refusing update from remote peer 5fbb448689c949d98dec8e37cf5c3478: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 14:00:14.159240 14410 consensus_queue.cc:1035] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:14.164595 14410 raft_consensus.cc:2932] T 03409b5f413f499c919f2c6a5a4126f7 P 5fbb448689c949d98dec8e37cf5c3478 [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 7 to 9, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:14.165449 13433 raft_consensus.cc:2932] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7 [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 7 to 9, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:14.168092 11624 catalog_manager.cc:5484] T 03409b5f413f499c919f2c6a5a4126f7 P 7553d1bab6f642ac8395c928da1c58d7 reported cstate change: config changed from index 7 to 9, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New cstate: current_term: 3 leader_uuid: "5fbb448689c949d98dec8e37cf5c3478" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false } } }
I20240502 14:00:14.168774 11612 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 03409b5f413f499c919f2c6a5a4126f7 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:14.175292 11747 tablet_service.cc:1514] Processing DeleteTablet for tablet 03409b5f413f499c919f2c6a5a4126f7 with delete_type TABLET_DATA_TOMBSTONED (TS 73ce3d001cdf4d0c8e1487958b7694ea not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43190
I20240502 14:00:14.179510 14416 tablet_replica.cc:330] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea: stopping tablet replica
I20240502 14:00:14.179730 14416 raft_consensus.cc:2229] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:14.180430 14416 raft_consensus.cc:2258] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:14.183012 14416 ts_tablet_manager.cc:1875] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:14.198889 14416 ts_tablet_manager.cc:1888] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.8
I20240502 14:00:14.198995 14416 log.cc:1196] T 03409b5f413f499c919f2c6a5a4126f7 P 73ce3d001cdf4d0c8e1487958b7694ea: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/wals/03409b5f413f499c919f2c6a5a4126f7
I20240502 14:00:14.201189 11612 catalog_manager.cc:4830] TS 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411): tablet 03409b5f413f499c919f2c6a5a4126f7 (table TestTable [id=977ef0b76afd4ccc955a0f354e4f5725]) successfully deleted
I20240502 14:00:14.653942 12008 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:14.658713 13020 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:14.665369 11877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:14.672441 13151 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:14.687474 13283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:14.687484 11747 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:14.694171 13414 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:15.087167 13020 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:15.089148 11747 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:15.093858 13283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:15.114918 12008 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:15.144004 13414 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:15.190809 13151 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:15.224283 11877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:15.943454 13020 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:15.960013 11747 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:16.008749 12008 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:16.024217 11877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:16.039343 13414 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:16.050508 13283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:16.083130 13151 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:16.417232 13151 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:16.455139 12008 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:16.487912 13414 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:16.491133 13283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:16.499810 13020 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:16.527750 11747 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:16.549743 11877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:16.706347 13151 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:16.707589 13020 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:16.710330 13414 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:16.711529 11877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:16.717037 13283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:16.718519 11747 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:16.724517 12008 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
12453ce20bf34d93b25ce6a40cfe613c | 127.8.86.190:44201 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.8.86.148:40301 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
4f6f2d2b1569415cbbbea2466dc28e28 | 127.8.86.132:42709 | HEALTHY | <none> | 0 | 0
5fbb448689c949d98dec8e37cf5c3478 | 127.8.86.133:44919 | HEALTHY | <none> | 1 | 0
65a078666d30488f9c204d93e6394ef9 | 127.8.86.131:45325 | HEALTHY | <none> | 3 | 0
73ce3d001cdf4d0c8e1487958b7694ea | 127.8.86.129:39411 | HEALTHY | <none> | 4 | 0
7553d1bab6f642ac8395c928da1c58d7 | 127.8.86.135:34019 | HEALTHY | <none> | 1 | 0
a36d5a88b1a34006bd3d040f5a715766 | 127.8.86.134:36541 | HEALTHY | <none> | 2 | 0
cbf92f56d7144a22a6d482c73c7c7ea9 | 127.8.86.130:44377 | HEALTHY | <none> | 5 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.8.86.129 | experimental | 127.8.86.129:39411
local_ip_for_outbound_sockets | 127.8.86.130 | experimental | 127.8.86.130:44377
local_ip_for_outbound_sockets | 127.8.86.131 | experimental | 127.8.86.131:45325
local_ip_for_outbound_sockets | 127.8.86.132 | experimental | 127.8.86.132:42709
local_ip_for_outbound_sockets | 127.8.86.133 | experimental | 127.8.86.133:44919
local_ip_for_outbound_sockets | 127.8.86.134 | experimental | 127.8.86.134:36541
local_ip_for_outbound_sockets | 127.8.86.135 | experimental | 127.8.86.135:34019
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.8.86.129:39411
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.8.86.130:44377
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.8.86.131:45325
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.8.86.132:42709
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.8.86.133:44919
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.8.86.134:36541
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.8.86.135:34019
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.8.86.148:40301 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 14:00:16.766026 8538 log_verifier.cc:126] Checking tablet 03409b5f413f499c919f2c6a5a4126f7
I20240502 14:00:16.782092 8538 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 03409b5f413f499c919f2c6a5a4126f7
I20240502 14:00:16.782160 8538 log_verifier.cc:126] Checking tablet 11f585c28308464bb46aabdf0ad98a08
I20240502 14:00:16.843005 8538 log_verifier.cc:177] Verified matching terms for 302 ops in tablet 11f585c28308464bb46aabdf0ad98a08
I20240502 14:00:16.843153 8538 log_verifier.cc:126] Checking tablet 182302167a0541888cb99bcd66635125
I20240502 14:00:16.915853 8538 log_verifier.cc:177] Verified matching terms for 338 ops in tablet 182302167a0541888cb99bcd66635125
I20240502 14:00:16.916013 8538 log_verifier.cc:126] Checking tablet 1870fb0c459f49cab36812d16e30ddf5
I20240502 14:00:16.975572 8538 log_verifier.cc:177] Verified matching terms for 320 ops in tablet 1870fb0c459f49cab36812d16e30ddf5
I20240502 14:00:16.975692 8538 log_verifier.cc:126] Checking tablet 236933482b6b41cf9e84a2eeb3b57638
I20240502 14:00:17.035373 8538 log_verifier.cc:177] Verified matching terms for 330 ops in tablet 236933482b6b41cf9e84a2eeb3b57638
I20240502 14:00:17.035522 8538 log_verifier.cc:126] Checking tablet 259e249430fa41a2bc2528b65d022b67
I20240502 14:00:17.091096 8538 log_verifier.cc:177] Verified matching terms for 299 ops in tablet 259e249430fa41a2bc2528b65d022b67
I20240502 14:00:17.091223 8538 log_verifier.cc:126] Checking tablet 3a50cea2712d48ca812cea7983d16840
I20240502 14:00:17.150772 8538 log_verifier.cc:177] Verified matching terms for 302 ops in tablet 3a50cea2712d48ca812cea7983d16840
I20240502 14:00:17.150889 8538 log_verifier.cc:126] Checking tablet 49041719e0ae42a390452b8716cc3ca7
I20240502 14:00:17.212661 8538 log_verifier.cc:177] Verified matching terms for 323 ops in tablet 49041719e0ae42a390452b8716cc3ca7
I20240502 14:00:17.212806 8538 log_verifier.cc:126] Checking tablet 7cd1b1872364411e86b810420c54034a
I20240502 14:00:17.266160 8538 log_verifier.cc:177] Verified matching terms for 277 ops in tablet 7cd1b1872364411e86b810420c54034a
I20240502 14:00:17.266330 8538 log_verifier.cc:126] Checking tablet 986b926ead014774925e3b8a3892c56d
I20240502 14:00:17.319419 8538 log_verifier.cc:177] Verified matching terms for 286 ops in tablet 986b926ead014774925e3b8a3892c56d
I20240502 14:00:17.319562 8538 log_verifier.cc:126] Checking tablet b9a6f7fca259430dac2ca17a8b180b2a
I20240502 14:00:17.377673 8538 log_verifier.cc:177] Verified matching terms for 315 ops in tablet b9a6f7fca259430dac2ca17a8b180b2a
I20240502 14:00:17.377825 8538 log_verifier.cc:126] Checking tablet b9bc1c84cb144965a21bdc40f099416d
I20240502 14:00:17.436126 8538 log_verifier.cc:177] Verified matching terms for 311 ops in tablet b9bc1c84cb144965a21bdc40f099416d
I20240502 14:00:17.436281 8538 log_verifier.cc:126] Checking tablet c0230720456b49339caaa9d313b6ce64
I20240502 14:00:17.494478 8538 log_verifier.cc:177] Verified matching terms for 261 ops in tablet c0230720456b49339caaa9d313b6ce64
I20240502 14:00:17.494596 8538 log_verifier.cc:126] Checking tablet c951304cdaa54040a81e51d076fdeaaf
I20240502 14:00:17.551831 8538 log_verifier.cc:177] Verified matching terms for 298 ops in tablet c951304cdaa54040a81e51d076fdeaaf
I20240502 14:00:17.551955 8538 log_verifier.cc:126] Checking tablet db4ce5e277504649a781e23861baa667
I20240502 14:00:17.614447 8538 log_verifier.cc:177] Verified matching terms for 330 ops in tablet db4ce5e277504649a781e23861baa667
I20240502 14:00:17.614579 8538 log_verifier.cc:126] Checking tablet dc89c51bfacd43d7a7cca53bc5624438
I20240502 14:00:17.683904 8538 log_verifier.cc:177] Verified matching terms for 377 ops in tablet dc89c51bfacd43d7a7cca53bc5624438
I20240502 14:00:17.686061 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.136:0
--local_ip_for_outbound_sockets=127.8.86.136
--webserver_interface=127.8.86.136
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44201
--builtin_ntp_servers=127.8.86.148:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:00:17.795554 14642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:17.795773 14642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:17.795820 14642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:17.799167 14642 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:00:17.799230 14642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:17.799315 14642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.136
I20240502 14:00:17.803071 14642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.8.86.136
--webserver_port=0
--tserver_master_addrs=127.8.86.190:44201
--never_fsync=true
--heap_profile_path=/tmp/kudu.14642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.136
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:17.803929 14642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:17.805003 14642 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:17.813266 14647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:17.813422 14648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:17.814275 14642 server_base.cc:1034] running on GCE node
W20240502 14:00:17.814381 14650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:17.814816 14642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:17.815338 14642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:17.816483 14642 hybrid_clock.cc:648] HybridClock initialized: now 1714658417816472 us; error 25 us; skew 500 ppm
I20240502 14:00:17.821954 14642 webserver.cc:460] Webserver started at http://127.8.86.136:36191/ using document root <none> and password file <none>
I20240502 14:00:17.822531 14642 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:17.822589 14642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:17.822780 14642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:17.824602 14642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c"
format_stamp: "Formatted at 2024-05-02 14:00:17 on dist-test-slave-h6v0"
I20240502 14:00:17.829746 14642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c"
format_stamp: "Formatted at 2024-05-02 14:00:17 on dist-test-slave-h6v0"
I20240502 14:00:17.833331 14642 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:00:17.835840 14656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:17.836884 14642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:17.837009 14642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c"
format_stamp: "Formatted at 2024-05-02 14:00:17 on dist-test-slave-h6v0"
I20240502 14:00:17.837100 14642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:17.853164 14642 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:17.853758 14642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:17.853909 14642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:17.854447 14642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:17.855306 14642 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:17.855358 14642 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:17.855415 14642 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:17.855445 14642 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:17.864163 14642 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.136:36081
I20240502 14:00:17.864198 14769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.136:36081 every 8 connection(s)
I20240502 14:00:17.865046 14642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:00:17.872202 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 14642
I20240502 14:00:17.872332 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 14:00:17.873670 14770 heartbeater.cc:347] Connected to a master server at 127.8.86.190:44201
I20240502 14:00:17.873888 14770 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:17.874433 14770 heartbeater.cc:511] Master 127.8.86.190:44201 requested a full tablet report, sending...
I20240502 14:00:17.875452 11622 ts_manager.cc:194] Registered new tserver with Master: 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136:36081)
I20240502 14:00:17.876000 11622 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.136:42499
I20240502 14:00:18.064160 13020 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:18.064162 12008 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:18.069116 14704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:18.076666 13414 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:18.083765 11747 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:18.087466 11877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:18.101985 13283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:18.109617 13151 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:18.191628 14704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:18.194326 13151 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:18.198498 12008 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:18.205348 11747 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:18.205613 13414 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:18.210464 11877 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:18.210673 13283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:18.235806 13020 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:18.267931 12026 consensus_queue.cc:237] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 320, Committed index: 320, Last appended: 1.320, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } }
I20240502 14:00:18.269248 13037 raft_consensus.cc:1261] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 1 index: 320. Preceding OpId from leader: term: 1 index: 321. (index mismatch)
I20240502 14:00:18.269640 14868 consensus_queue.cc:1035] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 321, Last known committed idx: 320, Time since last communication: 0.000s
I20240502 14:00:18.271323 14596 raft_consensus.cc:2932] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Committing config change with OpId 1.321: config changed from index 45 to 321, NON_VOTER 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) added. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } } }
I20240502 14:00:18.272918 13037 raft_consensus.cc:2932] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Committing config change with OpId 1.321: config changed from index 45 to 321, NON_VOTER 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) added. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } } }
W20240502 14:00:18.273666 11964 consensus_peers.cc:485] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 -> Peer 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136:36081): Couldn't send request to peer 1ca55a1c9fcb4a1487dcde45aec09e3c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1870fb0c459f49cab36812d16e30ddf5. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:18.274171 11619 catalog_manager.cc:5484] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 reported cstate change: config changed from index 45 to 321, NON_VOTER 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) added. New cstate: current_term: 1 leader_uuid: "65a078666d30488f9c204d93e6394ef9" committed_config { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:18.281093 13433 consensus_queue.cc:237] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 311, Committed index: 311, Last appended: 2.311, Last appended by leader: 158, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } }
I20240502 14:00:18.282485 13172 raft_consensus.cc:1261] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Refusing update from remote peer 7553d1bab6f642ac8395c928da1c58d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 311. Preceding OpId from leader: term: 2 index: 312. (index mismatch)
I20240502 14:00:18.282980 14860 consensus_queue.cc:1035] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 312, Last known committed idx: 311, Time since last communication: 0.000s
I20240502 14:00:18.284696 14858 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 2 LEADER]: Committing config change with OpId 2.312: config changed from index 162 to 312, NON_VOTER 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) added. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } } }
I20240502 14:00:18.285717 13172 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Committing config change with OpId 2.312: config changed from index 162 to 312, NON_VOTER 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) added. New config: { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } } }
W20240502 14:00:18.286013 13368 consensus_peers.cc:485] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 -> Peer 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136:36081): Couldn't send request to peer 1ca55a1c9fcb4a1487dcde45aec09e3c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9bc1c84cb144965a21bdc40f099416d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:18.287173 11617 catalog_manager.cc:5484] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 reported cstate change: config changed from index 162 to 312, NON_VOTER 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) added. New cstate: current_term: 2 leader_uuid: "7553d1bab6f642ac8395c928da1c58d7" committed_config { opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:18.294476 11768 consensus_queue.cc:237] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 1.302, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } }
I20240502 14:00:18.296144 13037 raft_consensus.cc:1261] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch)
I20240502 14:00:18.296689 14890 consensus_queue.cc:1035] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20240502 14:00:18.298529 13699 raft_consensus.cc:2932] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Committing config change with OpId 1.303: config changed from index 79 to 303, NON_VOTER 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) added. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } } }
I20240502 14:00:18.299016 13037 raft_consensus.cc:2932] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 79 to 303, NON_VOTER 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) added. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } } }
W20240502 14:00:18.299973 11702 consensus_peers.cc:485] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea -> Peer 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136:36081): Couldn't send request to peer 1ca55a1c9fcb4a1487dcde45aec09e3c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11f585c28308464bb46aabdf0ad98a08. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:18.301585 11617 catalog_manager.cc:5484] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: config changed from index 79 to 303, NON_VOTER 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) added. New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:18.307745 11899 consensus_queue.cc:237] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 2.302, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } }
I20240502 14:00:18.309096 13172 raft_consensus.cc:1261] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 2 index: 302. Preceding OpId from leader: term: 2 index: 303. (index mismatch)
I20240502 14:00:18.309494 14899 consensus_queue.cc:1035] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20240502 14:00:18.311285 14123 raft_consensus.cc:2932] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 LEADER]: Committing config change with OpId 2.303: config changed from index 108 to 303, NON_VOTER 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) added. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } } }
I20240502 14:00:18.311645 13172 raft_consensus.cc:2932] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Committing config change with OpId 2.303: config changed from index 108 to 303, NON_VOTER 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) added. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } } }
W20240502 14:00:18.313697 11833 consensus_peers.cc:485] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 -> Peer 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136:36081): Couldn't send request to peer 1ca55a1c9fcb4a1487dcde45aec09e3c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a50cea2712d48ca812cea7983d16840. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:18.313895 11619 catalog_manager.cc:5484] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 reported cstate change: config changed from index 108 to 303, NON_VOTER 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) added. New cstate: current_term: 2 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } } }
I20240502 14:00:18.721635 14942 ts_tablet_manager.cc:902] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c: Initiating tablet copy from peer 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325)
I20240502 14:00:18.722380 14942 tablet_copy_client.cc:308] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:45325
I20240502 14:00:18.725847 12050 tablet_copy_service.cc:140] P 65a078666d30488f9c204d93e6394ef9: Received BeginTabletCopySession request for tablet 1870fb0c459f49cab36812d16e30ddf5 from peer 1ca55a1c9fcb4a1487dcde45aec09e3c ({username='slave'} at 127.8.86.136:35223)
I20240502 14:00:18.725950 12050 tablet_copy_service.cc:161] P 65a078666d30488f9c204d93e6394ef9: Beginning new tablet copy session on tablet 1870fb0c459f49cab36812d16e30ddf5 from peer 1ca55a1c9fcb4a1487dcde45aec09e3c at {username='slave'} at 127.8.86.136:35223: session id = 1ca55a1c9fcb4a1487dcde45aec09e3c-1870fb0c459f49cab36812d16e30ddf5
I20240502 14:00:18.726661 12050 tablet_copy_source_session.cc:205] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:18.729573 14942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1870fb0c459f49cab36812d16e30ddf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.734043 14942 tablet_copy_client.cc:783] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:18.734210 14942 tablet_copy_client.cc:647] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:18.736853 14942 tablet_copy_client.cc:523] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:18.738442 14942 tablet_bootstrap.cc:492] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c: Bootstrap starting.
I20240502 14:00:18.747021 14948 ts_tablet_manager.cc:902] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c: Initiating tablet copy from peer 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411)
I20240502 14:00:18.747577 14948 tablet_copy_client.cc:308] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:39411
I20240502 14:00:18.751137 11788 tablet_copy_service.cc:140] P 73ce3d001cdf4d0c8e1487958b7694ea: Received BeginTabletCopySession request for tablet 11f585c28308464bb46aabdf0ad98a08 from peer 1ca55a1c9fcb4a1487dcde45aec09e3c ({username='slave'} at 127.8.86.136:36953)
I20240502 14:00:18.751256 11788 tablet_copy_service.cc:161] P 73ce3d001cdf4d0c8e1487958b7694ea: Beginning new tablet copy session on tablet 11f585c28308464bb46aabdf0ad98a08 from peer 1ca55a1c9fcb4a1487dcde45aec09e3c at {username='slave'} at 127.8.86.136:36953: session id = 1ca55a1c9fcb4a1487dcde45aec09e3c-11f585c28308464bb46aabdf0ad98a08
I20240502 14:00:18.752203 11788 tablet_copy_source_session.cc:205] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:18.752962 14948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f585c28308464bb46aabdf0ad98a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.754588 14948 tablet_copy_client.cc:783] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:18.754724 14948 tablet_copy_client.cc:647] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:18.756947 14948 tablet_copy_client.cc:523] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:18.758124 14948 tablet_bootstrap.cc:492] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c: Bootstrap starting.
I20240502 14:00:18.767614 14942 log.cc:824] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:18.781100 14975 ts_tablet_manager.cc:902] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c: Initiating tablet copy from peer cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377)
I20240502 14:00:18.782126 14975 tablet_copy_client.cc:308] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:44377
I20240502 14:00:18.785220 11919 tablet_copy_service.cc:140] P cbf92f56d7144a22a6d482c73c7c7ea9: Received BeginTabletCopySession request for tablet 3a50cea2712d48ca812cea7983d16840 from peer 1ca55a1c9fcb4a1487dcde45aec09e3c ({username='slave'} at 127.8.86.136:45281)
I20240502 14:00:18.785334 11919 tablet_copy_service.cc:161] P cbf92f56d7144a22a6d482c73c7c7ea9: Beginning new tablet copy session on tablet 3a50cea2712d48ca812cea7983d16840 from peer 1ca55a1c9fcb4a1487dcde45aec09e3c at {username='slave'} at 127.8.86.136:45281: session id = 1ca55a1c9fcb4a1487dcde45aec09e3c-3a50cea2712d48ca812cea7983d16840
I20240502 14:00:18.786404 11919 tablet_copy_source_session.cc:205] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:18.787276 14975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a50cea2712d48ca812cea7983d16840. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.789070 14982 ts_tablet_manager.cc:902] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c: Initiating tablet copy from peer 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135:34019)
I20240502 14:00:18.789295 14975 tablet_copy_client.cc:783] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:18.789428 14975 tablet_copy_client.cc:647] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:18.789825 14982 tablet_copy_client.cc:308] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.135:34019
I20240502 14:00:18.793747 14975 tablet_copy_client.cc:523] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:18.795006 14975 tablet_bootstrap.cc:492] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c: Bootstrap starting.
I20240502 14:00:18.795058 13454 tablet_copy_service.cc:140] P 7553d1bab6f642ac8395c928da1c58d7: Received BeginTabletCopySession request for tablet b9bc1c84cb144965a21bdc40f099416d from peer 1ca55a1c9fcb4a1487dcde45aec09e3c ({username='slave'} at 127.8.86.136:53893)
I20240502 14:00:18.795176 13454 tablet_copy_service.cc:161] P 7553d1bab6f642ac8395c928da1c58d7: Beginning new tablet copy session on tablet b9bc1c84cb144965a21bdc40f099416d from peer 1ca55a1c9fcb4a1487dcde45aec09e3c at {username='slave'} at 127.8.86.136:53893: session id = 1ca55a1c9fcb4a1487dcde45aec09e3c-b9bc1c84cb144965a21bdc40f099416d
I20240502 14:00:18.796617 13454 tablet_copy_source_session.cc:205] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:18.797694 14982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9bc1c84cb144965a21bdc40f099416d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:18.799650 14982 tablet_copy_client.cc:783] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:18.799790 14982 tablet_copy_client.cc:647] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:18.802078 14982 tablet_copy_client.cc:523] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:18.803241 14982 tablet_bootstrap.cc:492] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c: Bootstrap starting.
I20240502 14:00:18.877839 14770 heartbeater.cc:503] Master 127.8.86.190:44201 was elected leader, sending a full tablet report...
I20240502 14:00:19.003033 14942 tablet_bootstrap.cc:492] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=321 overwritten=0 applied=321 ignored=0} inserts{seen=949 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:19.003559 14942 tablet_bootstrap.cc:492] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c: Bootstrap complete.
I20240502 14:00:19.004022 14942 ts_tablet_manager.cc:1367] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c: Time spent bootstrapping tablet: real 0.266s user 0.225s sys 0.021s
I20240502 14:00:19.009050 14942 raft_consensus.cc:348] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } }
I20240502 14:00:19.009451 14942 raft_consensus.cc:729] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ca55a1c9fcb4a1487dcde45aec09e3c, State: Initialized, Role: LEARNER
I20240502 14:00:19.009861 14942 consensus_queue.cc:260] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 321, Last appended: 1.321, Last appended by leader: 321, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } }
I20240502 14:00:19.011242 14942 ts_tablet_manager.cc:1398] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240502 14:00:19.013753 12050 tablet_copy_service.cc:342] P 65a078666d30488f9c204d93e6394ef9: Request end of tablet copy session 1ca55a1c9fcb4a1487dcde45aec09e3c-1870fb0c459f49cab36812d16e30ddf5 received from {username='slave'} at 127.8.86.136:35223
I20240502 14:00:19.013845 12050 tablet_copy_service.cc:433] P 65a078666d30488f9c204d93e6394ef9: ending tablet copy session 1ca55a1c9fcb4a1487dcde45aec09e3c-1870fb0c459f49cab36812d16e30ddf5 on tablet 1870fb0c459f49cab36812d16e30ddf5 with peer 1ca55a1c9fcb4a1487dcde45aec09e3c
I20240502 14:00:19.038268 14948 tablet_bootstrap.cc:492] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=303 overwritten=0 applied=303 ignored=0} inserts{seen=972 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:19.038725 14948 tablet_bootstrap.cc:492] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c: Bootstrap complete.
I20240502 14:00:19.038923 14948 ts_tablet_manager.cc:1367] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c: Time spent bootstrapping tablet: real 0.281s user 0.219s sys 0.013s
I20240502 14:00:19.039397 14948 raft_consensus.cc:348] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } }
I20240502 14:00:19.039501 14948 raft_consensus.cc:729] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ca55a1c9fcb4a1487dcde45aec09e3c, State: Initialized, Role: LEARNER
I20240502 14:00:19.039624 14948 consensus_queue.cc:260] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 303, Last appended: 1.303, Last appended by leader: 303, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } }
I20240502 14:00:19.039884 14948 ts_tablet_manager.cc:1398] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.040640 11788 tablet_copy_service.cc:342] P 73ce3d001cdf4d0c8e1487958b7694ea: Request end of tablet copy session 1ca55a1c9fcb4a1487dcde45aec09e3c-11f585c28308464bb46aabdf0ad98a08 received from {username='slave'} at 127.8.86.136:36953
I20240502 14:00:19.040721 11788 tablet_copy_service.cc:433] P 73ce3d001cdf4d0c8e1487958b7694ea: ending tablet copy session 1ca55a1c9fcb4a1487dcde45aec09e3c-11f585c28308464bb46aabdf0ad98a08 on tablet 11f585c28308464bb46aabdf0ad98a08 with peer 1ca55a1c9fcb4a1487dcde45aec09e3c
I20240502 14:00:19.057273 14982 tablet_bootstrap.cc:492] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=312 overwritten=0 applied=312 ignored=0} inserts{seen=977 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:19.057624 14982 tablet_bootstrap.cc:492] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c: Bootstrap complete.
I20240502 14:00:19.057834 14982 ts_tablet_manager.cc:1367] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c: Time spent bootstrapping tablet: real 0.255s user 0.225s sys 0.008s
I20240502 14:00:19.057987 14975 tablet_bootstrap.cc:492] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=303 overwritten=0 applied=303 ignored=0} inserts{seen=984 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:19.058228 14975 tablet_bootstrap.cc:492] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c: Bootstrap complete.
I20240502 14:00:19.058322 14982 raft_consensus.cc:348] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } }
I20240502 14:00:19.058442 14975 ts_tablet_manager.cc:1367] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c: Time spent bootstrapping tablet: real 0.263s user 0.214s sys 0.018s
I20240502 14:00:19.058451 14982 raft_consensus.cc:729] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1ca55a1c9fcb4a1487dcde45aec09e3c, State: Initialized, Role: LEARNER
I20240502 14:00:19.058593 14982 consensus_queue.cc:260] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 312, Last appended: 2.312, Last appended by leader: 312, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 312 OBSOLETE_local: false peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } }
I20240502 14:00:19.058884 14982 ts_tablet_manager.cc:1398] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 14:00:19.058904 14975 raft_consensus.cc:348] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } }
I20240502 14:00:19.059005 14975 raft_consensus.cc:729] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1ca55a1c9fcb4a1487dcde45aec09e3c, State: Initialized, Role: LEARNER
I20240502 14:00:19.059145 14975 consensus_queue.cc:260] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 303, Last appended: 2.303, Last appended by leader: 303, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: true } }
I20240502 14:00:19.059406 14975 ts_tablet_manager.cc:1398] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.059484 13454 tablet_copy_service.cc:342] P 7553d1bab6f642ac8395c928da1c58d7: Request end of tablet copy session 1ca55a1c9fcb4a1487dcde45aec09e3c-b9bc1c84cb144965a21bdc40f099416d received from {username='slave'} at 127.8.86.136:53893
I20240502 14:00:19.059590 13454 tablet_copy_service.cc:433] P 7553d1bab6f642ac8395c928da1c58d7: ending tablet copy session 1ca55a1c9fcb4a1487dcde45aec09e3c-b9bc1c84cb144965a21bdc40f099416d on tablet b9bc1c84cb144965a21bdc40f099416d with peer 1ca55a1c9fcb4a1487dcde45aec09e3c
I20240502 14:00:19.060230 11919 tablet_copy_service.cc:342] P cbf92f56d7144a22a6d482c73c7c7ea9: Request end of tablet copy session 1ca55a1c9fcb4a1487dcde45aec09e3c-3a50cea2712d48ca812cea7983d16840 received from {username='slave'} at 127.8.86.136:45281
I20240502 14:00:19.060309 11919 tablet_copy_service.cc:433] P cbf92f56d7144a22a6d482c73c7c7ea9: ending tablet copy session 1ca55a1c9fcb4a1487dcde45aec09e3c-3a50cea2712d48ca812cea7983d16840 on tablet 3a50cea2712d48ca812cea7983d16840 with peer 1ca55a1c9fcb4a1487dcde45aec09e3c
I20240502 14:00:19.227519 14724 raft_consensus.cc:1203] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.311->[2.312-2.312] Dedup: 2.312->[]
I20240502 14:00:19.243706 14724 raft_consensus.cc:1203] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.302->[1.303-1.303] Dedup: 1.303->[]
I20240502 14:00:19.297250 14724 raft_consensus.cc:1203] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.320->[1.321-1.321] Dedup: 1.321->[]
I20240502 14:00:19.342510 14724 raft_consensus.cc:1203] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.302->[2.303-2.303] Dedup: 2.303->[]
I20240502 14:00:19.696110 14890 raft_consensus.cc:1050] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea: attempting to promote NON_VOTER 1ca55a1c9fcb4a1487dcde45aec09e3c to VOTER
I20240502 14:00:19.696524 14890 consensus_queue.cc:237] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 303, Committed index: 303, Last appended: 1.303, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:19.697947 13037 raft_consensus.cc:1261] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 303. Preceding OpId from leader: term: 1 index: 304. (index mismatch)
I20240502 14:00:19.698105 14724 raft_consensus.cc:1261] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 1 LEARNER]: Refusing update from remote peer 73ce3d001cdf4d0c8e1487958b7694ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 303. Preceding OpId from leader: term: 1 index: 304. (index mismatch)
I20240502 14:00:19.698503 14890 consensus_queue.cc:1035] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I20240502 14:00:19.698685 15017 consensus_queue.cc:1035] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I20240502 14:00:19.700505 15063 raft_consensus.cc:2932] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Committing config change with OpId 1.304: config changed from index 303 to 304, 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:19.700930 13037 raft_consensus.cc:2932] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Committing config change with OpId 1.304: config changed from index 303 to 304, 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:19.702050 15011 raft_consensus.cc:1050] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7: attempting to promote NON_VOTER 1ca55a1c9fcb4a1487dcde45aec09e3c to VOTER
I20240502 14:00:19.702533 15011 consensus_queue.cc:237] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 312, Committed index: 312, Last appended: 2.312, Last appended by leader: 158, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:19.703320 11623 catalog_manager.cc:5484] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea reported cstate change: config changed from index 303 to 304, 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" committed_config { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:19.704224 14724 raft_consensus.cc:1261] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 LEARNER]: Refusing update from remote peer 7553d1bab6f642ac8395c928da1c58d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 312. Preceding OpId from leader: term: 2 index: 313. (index mismatch)
I20240502 14:00:19.704463 13172 raft_consensus.cc:1261] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Refusing update from remote peer 7553d1bab6f642ac8395c928da1c58d7: Log matching property violated. Preceding OpId in replica: term: 2 index: 312. Preceding OpId from leader: term: 2 index: 313. (index mismatch)
I20240502 14:00:19.704494 14723 raft_consensus.cc:2932] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 1 FOLLOWER]: Committing config change with OpId 1.304: config changed from index 303 to 304, 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:19.705281 15010 consensus_queue.cc:1035] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 313, Last known committed idx: 312, Time since last communication: 0.000s
I20240502 14:00:19.705871 15010 consensus_queue.cc:1035] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 313, Last known committed idx: 312, Time since last communication: 0.000s
I20240502 14:00:19.708447 14980 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 2 LEADER]: Committing config change with OpId 2.313: config changed from index 312 to 313, 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:19.708886 14723 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 FOLLOWER]: Committing config change with OpId 2.313: config changed from index 312 to 313, 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:19.711146 11619 catalog_manager.cc:5484] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 reported cstate change: config changed from index 312 to 313, 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7553d1bab6f642ac8395c928da1c58d7" committed_config { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:19.714354 13172 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Committing config change with OpId 2.313: config changed from index 312 to 313, 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:19.729954 11768 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "11f585c28308464bb46aabdf0ad98a08"
dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39678
I20240502 14:00:19.730094 11768 raft_consensus.cc:595] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:19.734124 15033 raft_consensus.cc:1050] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9: attempting to promote NON_VOTER 1ca55a1c9fcb4a1487dcde45aec09e3c to VOTER
I20240502 14:00:19.734566 15033 consensus_queue.cc:237] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 1.321, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:19.735849 14723 raft_consensus.cc:1261] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 1 LEARNER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 1 index: 321. Preceding OpId from leader: term: 1 index: 322. (index mismatch)
I20240502 14:00:19.735850 13037 raft_consensus.cc:1261] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Refusing update from remote peer 65a078666d30488f9c204d93e6394ef9: Log matching property violated. Preceding OpId in replica: term: 1 index: 321. Preceding OpId from leader: term: 1 index: 322. (index mismatch)
I20240502 14:00:19.736282 15033 consensus_queue.cc:1035] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.000s
I20240502 14:00:19.736485 15034 consensus_queue.cc:1035] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.000s
I20240502 14:00:19.738155 15033 raft_consensus.cc:2932] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Committing config change with OpId 1.322: config changed from index 321 to 322, 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:19.738615 14723 raft_consensus.cc:2932] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 1 FOLLOWER]: Committing config change with OpId 1.322: config changed from index 321 to 322, 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:19.739570 13037 raft_consensus.cc:2932] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Committing config change with OpId 1.322: config changed from index 321 to 322, 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:19.741784 11617 catalog_manager.cc:5484] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 reported cstate change: config changed from index 321 to 322, 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "65a078666d30488f9c204d93e6394ef9" committed_config { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:19.744148 13433 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b9bc1c84cb144965a21bdc40f099416d"
dest_uuid: "7553d1bab6f642ac8395c928da1c58d7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48090
I20240502 14:00:19.744333 13433 raft_consensus.cc:595] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 2 LEADER]: Received request to transfer leadership
I20240502 14:00:19.926502 15036 raft_consensus.cc:1050] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9: attempting to promote NON_VOTER 1ca55a1c9fcb4a1487dcde45aec09e3c to VOTER
I20240502 14:00:19.926853 15036 consensus_queue.cc:237] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 303, Committed index: 303, Last appended: 2.303, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:19.927980 14723 raft_consensus.cc:1261] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 LEARNER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 2 index: 303. Preceding OpId from leader: term: 2 index: 304. (index mismatch)
I20240502 14:00:19.928068 13172 raft_consensus.cc:1261] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Refusing update from remote peer cbf92f56d7144a22a6d482c73c7c7ea9: Log matching property violated. Preceding OpId in replica: term: 2 index: 303. Preceding OpId from leader: term: 2 index: 304. (index mismatch)
I20240502 14:00:19.928382 15105 consensus_queue.cc:1035] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I20240502 14:00:19.928591 15036 consensus_queue.cc:1035] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I20240502 14:00:19.930466 15103 raft_consensus.cc:2932] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 LEADER]: Committing config change with OpId 2.304: config changed from index 303 to 304, 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:19.930820 14723 raft_consensus.cc:2932] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 FOLLOWER]: Committing config change with OpId 2.304: config changed from index 303 to 304, 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:19.931023 13172 raft_consensus.cc:2932] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Committing config change with OpId 2.304: config changed from index 303 to 304, 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:19.933416 11619 catalog_manager.cc:5484] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c reported cstate change: config changed from index 303 to 304, 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" committed_config { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:19.954736 11899 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3a50cea2712d48ca812cea7983d16840"
dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41484
I20240502 14:00:19.954864 11899 raft_consensus.cc:595] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 LEADER]: Received request to transfer leadership
I20240502 14:00:19.964576 12026 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1870fb0c459f49cab36812d16e30ddf5"
dest_uuid: "65a078666d30488f9c204d93e6394ef9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37090
I20240502 14:00:19.964709 12026 raft_consensus.cc:595] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:19.974081 11768 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "11f585c28308464bb46aabdf0ad98a08"
dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39704
I20240502 14:00:19.974210 11768 raft_consensus.cc:595] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:19.984305 13433 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b9bc1c84cb144965a21bdc40f099416d"
dest_uuid: "7553d1bab6f642ac8395c928da1c58d7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48110
I20240502 14:00:19.984439 13433 raft_consensus.cc:595] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 2 LEADER]: Received request to transfer leadership
I20240502 14:00:20.084430 14890 raft_consensus.cc:979] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea: : Instructing follower 4f6f2d2b1569415cbbbea2466dc28e28 to start an election
I20240502 14:00:20.084559 14890 raft_consensus.cc:1067] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Signalling peer 4f6f2d2b1569415cbbbea2466dc28e28 to start an election
I20240502 14:00:20.085057 13037 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "11f585c28308464bb46aabdf0ad98a08"
dest_uuid: "4f6f2d2b1569415cbbbea2466dc28e28"
from {username='slave'} at 127.8.86.129:45645
I20240502 14:00:20.085227 13037 raft_consensus.cc:482] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:20.085281 13037 raft_consensus.cc:3037] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:20.086117 13037 raft_consensus.cc:504] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:20.087045 13037 leader_election.cc:288] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [CANDIDATE]: Term 2 election: Requested vote from peers 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411), 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136:36081)
I20240502 14:00:20.087437 11768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f585c28308464bb46aabdf0ad98a08" candidate_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" candidate_term: 2 candidate_status { last_received { term: 1 index: 304 } } ignore_live_leader: true dest_uuid: "73ce3d001cdf4d0c8e1487958b7694ea"
I20240502 14:00:20.087581 11768 raft_consensus.cc:3032] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:20.087633 11768 raft_consensus.cc:729] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 73ce3d001cdf4d0c8e1487958b7694ea, State: Running, Role: LEADER
I20240502 14:00:20.087786 11768 consensus_queue.cc:260] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 304, Committed index: 304, Last appended: 1.304, Last appended by leader: 304, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:20.087945 11768 raft_consensus.cc:3037] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:20.088946 11768 raft_consensus.cc:2454] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f6f2d2b1569415cbbbea2466dc28e28 in term 2.
I20240502 14:00:20.089325 12975 leader_election.cc:302] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f6f2d2b1569415cbbbea2466dc28e28, 73ce3d001cdf4d0c8e1487958b7694ea; no voters:
I20240502 14:00:20.089560 15067 raft_consensus.cc:2781] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:20.089752 15067 raft_consensus.cc:686] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 2 LEADER]: Becoming Leader. State: Replica: 4f6f2d2b1569415cbbbea2466dc28e28, State: Running, Role: LEADER
I20240502 14:00:20.090047 15067 consensus_queue.cc:237] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 304, Committed index: 304, Last appended: 1.304, Last appended by leader: 304, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:20.091203 14723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f585c28308464bb46aabdf0ad98a08" candidate_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" candidate_term: 2 candidate_status { last_received { term: 1 index: 304 } } ignore_live_leader: true dest_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c"
I20240502 14:00:20.091388 14723 raft_consensus.cc:3037] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:20.092196 14723 raft_consensus.cc:2454] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f6f2d2b1569415cbbbea2466dc28e28 in term 2.
I20240502 14:00:20.092273 11620 catalog_manager.cc:5484] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 reported cstate change: term changed from 1 to 2, leader changed from 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) to 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132). New cstate: current_term: 2 leader_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" committed_config { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:20.106833 15010 raft_consensus.cc:979] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7: : Instructing follower 5fbb448689c949d98dec8e37cf5c3478 to start an election
I20240502 14:00:20.106954 15010 raft_consensus.cc:1067] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 2 LEADER]: Signalling peer 5fbb448689c949d98dec8e37cf5c3478 to start an election
I20240502 14:00:20.107470 13172 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9bc1c84cb144965a21bdc40f099416d"
dest_uuid: "5fbb448689c949d98dec8e37cf5c3478"
from {username='slave'} at 127.8.86.135:55153
I20240502 14:00:20.107601 13172 raft_consensus.cc:482] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:20.107650 13172 raft_consensus.cc:3037] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:00:20.108413 13172 raft_consensus.cc:504] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:20.108821 13172 leader_election.cc:288] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [CANDIDATE]: Term 3 election: Requested vote from peers 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135:34019), 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136:36081)
I20240502 14:00:20.109148 13433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9bc1c84cb144965a21bdc40f099416d" candidate_uuid: "5fbb448689c949d98dec8e37cf5c3478" candidate_term: 3 candidate_status { last_received { term: 2 index: 313 } } ignore_live_leader: true dest_uuid: "7553d1bab6f642ac8395c928da1c58d7"
I20240502 14:00:20.109284 13433 raft_consensus.cc:3032] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:00:20.109318 13433 raft_consensus.cc:729] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7553d1bab6f642ac8395c928da1c58d7, State: Running, Role: LEADER
I20240502 14:00:20.109480 13433 consensus_queue.cc:260] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 313, Committed index: 313, Last appended: 2.313, Last appended by leader: 313, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:20.109684 13433 raft_consensus.cc:3037] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:00:20.110700 13433 raft_consensus.cc:2454] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fbb448689c949d98dec8e37cf5c3478 in term 3.
I20240502 14:00:20.111060 13107 leader_election.cc:302] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fbb448689c949d98dec8e37cf5c3478, 7553d1bab6f642ac8395c928da1c58d7; no voters:
I20240502 14:00:20.111236 15057 raft_consensus.cc:2781] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:00:20.111478 15057 raft_consensus.cc:686] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 3 LEADER]: Becoming Leader. State: Replica: 5fbb448689c949d98dec8e37cf5c3478, State: Running, Role: LEADER
I20240502 14:00:20.111656 15057 consensus_queue.cc:237] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 313, Committed index: 313, Last appended: 2.313, Last appended by leader: 313, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:20.112790 14723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9bc1c84cb144965a21bdc40f099416d" candidate_uuid: "5fbb448689c949d98dec8e37cf5c3478" candidate_term: 3 candidate_status { last_received { term: 2 index: 313 } } ignore_live_leader: true dest_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c"
I20240502 14:00:20.112896 14723 raft_consensus.cc:3037] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:00:20.113626 14723 raft_consensus.cc:2454] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fbb448689c949d98dec8e37cf5c3478 in term 3.
I20240502 14:00:20.114279 11620 catalog_manager.cc:5484] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 reported cstate change: term changed from 2 to 3, leader changed from 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) to 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133). New cstate: current_term: 3 leader_uuid: "5fbb448689c949d98dec8e37cf5c3478" committed_config { opid_index: 313 OBSOLETE_local: false peers { permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:20.162784 14868 raft_consensus.cc:979] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9: : Instructing follower 1ca55a1c9fcb4a1487dcde45aec09e3c to start an election
I20240502 14:00:20.162902 14868 raft_consensus.cc:1067] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Signalling peer 1ca55a1c9fcb4a1487dcde45aec09e3c to start an election
I20240502 14:00:20.163414 14723 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1870fb0c459f49cab36812d16e30ddf5"
dest_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c"
from {username='slave'} at 127.8.86.131:56903
I20240502 14:00:20.163630 14723 raft_consensus.cc:482] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:20.163702 14723 raft_consensus.cc:3037] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:20.164525 14723 raft_consensus.cc:504] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:20.165522 14723 leader_election.cc:288] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c [CANDIDATE]: Term 2 election: Requested vote from peers 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325), 4f6f2d2b1569415cbbbea2466dc28e28 (127.8.86.132:42709)
I20240502 14:00:20.165831 12026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1870fb0c459f49cab36812d16e30ddf5" candidate_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 322 } } ignore_live_leader: true dest_uuid: "65a078666d30488f9c204d93e6394ef9"
I20240502 14:00:20.165984 12026 raft_consensus.cc:3032] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:20.166051 12026 raft_consensus.cc:729] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 65a078666d30488f9c204d93e6394ef9, State: Running, Role: LEADER
I20240502 14:00:20.166218 12026 consensus_queue.cc:260] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 322, Committed index: 322, Last appended: 1.322, Last appended by leader: 322, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:20.166501 12026 raft_consensus.cc:3037] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:20.167593 12026 raft_consensus.cc:2454] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ca55a1c9fcb4a1487dcde45aec09e3c in term 2.
I20240502 14:00:20.167986 14659 leader_election.cc:302] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ca55a1c9fcb4a1487dcde45aec09e3c, 65a078666d30488f9c204d93e6394ef9; no voters:
I20240502 14:00:20.168207 14998 raft_consensus.cc:2781] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:20.168428 14998 raft_consensus.cc:686] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 LEADER]: Becoming Leader. State: Replica: 1ca55a1c9fcb4a1487dcde45aec09e3c, State: Running, Role: LEADER
I20240502 14:00:20.168761 14998 consensus_queue.cc:237] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 322, Committed index: 322, Last appended: 1.322, Last appended by leader: 322, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:20.169963 13037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1870fb0c459f49cab36812d16e30ddf5" candidate_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 322 } } ignore_live_leader: true dest_uuid: "4f6f2d2b1569415cbbbea2466dc28e28"
I20240502 14:00:20.170099 13037 raft_consensus.cc:3037] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:20.171018 13037 raft_consensus.cc:2454] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ca55a1c9fcb4a1487dcde45aec09e3c in term 2.
I20240502 14:00:20.171339 11620 catalog_manager.cc:5484] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c reported cstate change: term changed from 1 to 2, leader changed from 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) to 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136). New cstate: current_term: 2 leader_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" committed_config { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:20.195065 11899 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3a50cea2712d48ca812cea7983d16840"
dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41506
I20240502 14:00:20.195211 11899 raft_consensus.cc:595] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 LEADER]: Received request to transfer leadership
I20240502 14:00:20.403990 15103 raft_consensus.cc:979] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9: : Instructing follower 5fbb448689c949d98dec8e37cf5c3478 to start an election
I20240502 14:00:20.404127 15103 raft_consensus.cc:1067] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 LEADER]: Signalling peer 5fbb448689c949d98dec8e37cf5c3478 to start an election
I20240502 14:00:20.404577 13172 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3a50cea2712d48ca812cea7983d16840"
dest_uuid: "5fbb448689c949d98dec8e37cf5c3478"
from {username='slave'} at 127.8.86.130:54263
I20240502 14:00:20.404711 13172 raft_consensus.cc:482] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:20.404783 13172 raft_consensus.cc:3037] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:00:20.405611 13172 raft_consensus.cc:504] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:20.406051 13172 leader_election.cc:288] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [CANDIDATE]: Term 3 election: Requested vote from peers cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377), 1ca55a1c9fcb4a1487dcde45aec09e3c (127.8.86.136:36081)
I20240502 14:00:20.406386 11899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a50cea2712d48ca812cea7983d16840" candidate_uuid: "5fbb448689c949d98dec8e37cf5c3478" candidate_term: 3 candidate_status { last_received { term: 2 index: 304 } } ignore_live_leader: true dest_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9"
I20240502 14:00:20.406527 11899 raft_consensus.cc:3032] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:00:20.406582 11899 raft_consensus.cc:729] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cbf92f56d7144a22a6d482c73c7c7ea9, State: Running, Role: LEADER
I20240502 14:00:20.406533 14723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a50cea2712d48ca812cea7983d16840" candidate_uuid: "5fbb448689c949d98dec8e37cf5c3478" candidate_term: 3 candidate_status { last_received { term: 2 index: 304 } } ignore_live_leader: true dest_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c"
I20240502 14:00:20.406693 14723 raft_consensus.cc:3037] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:00:20.406736 11899 consensus_queue.cc:260] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 304, Committed index: 304, Last appended: 2.304, Last appended by leader: 304, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:20.406914 11899 raft_consensus.cc:3037] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:00:20.407497 14723 raft_consensus.cc:2454] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fbb448689c949d98dec8e37cf5c3478 in term 3.
I20240502 14:00:20.407727 11899 raft_consensus.cc:2454] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fbb448689c949d98dec8e37cf5c3478 in term 3.
I20240502 14:00:20.407837 13105 leader_election.cc:302] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ca55a1c9fcb4a1487dcde45aec09e3c, 5fbb448689c949d98dec8e37cf5c3478; no voters:
I20240502 14:00:20.408031 15057 raft_consensus.cc:2781] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:00:20.408114 15057 raft_consensus.cc:686] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 3 LEADER]: Becoming Leader. State: Replica: 5fbb448689c949d98dec8e37cf5c3478, State: Running, Role: LEADER
I20240502 14:00:20.408253 15057 consensus_queue.cc:237] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 304, Committed index: 304, Last appended: 2.304, Last appended by leader: 304, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:20.409718 11620 catalog_manager.cc:5484] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 reported cstate change: term changed from 2 to 3, leader changed from cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) to 5fbb448689c949d98dec8e37cf5c3478 (127.8.86.133). New cstate: current_term: 3 leader_uuid: "5fbb448689c949d98dec8e37cf5c3478" committed_config { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:20.479918 11768 raft_consensus.cc:1261] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Refusing update from remote peer 4f6f2d2b1569415cbbbea2466dc28e28: Log matching property violated. Preceding OpId in replica: term: 1 index: 304. Preceding OpId from leader: term: 2 index: 305. (index mismatch)
I20240502 14:00:20.480470 15067 consensus_queue.cc:1035] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73ce3d001cdf4d0c8e1487958b7694ea" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 39411 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 304, Time since last communication: 0.000s
I20240502 14:00:20.483464 14723 raft_consensus.cc:1261] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 FOLLOWER]: Refusing update from remote peer 4f6f2d2b1569415cbbbea2466dc28e28: Log matching property violated. Preceding OpId in replica: term: 1 index: 304. Preceding OpId from leader: term: 2 index: 305. (index mismatch)
I20240502 14:00:20.483899 15067 consensus_queue.cc:1035] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 304, Time since last communication: 0.000s
I20240502 14:00:20.614601 14723 raft_consensus.cc:1261] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 3 FOLLOWER]: Refusing update from remote peer 5fbb448689c949d98dec8e37cf5c3478: Log matching property violated. Preceding OpId in replica: term: 2 index: 313. Preceding OpId from leader: term: 3 index: 314. (index mismatch)
I20240502 14:00:20.615068 15057 consensus_queue.cc:1035] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
I20240502 14:00:20.618834 13433 raft_consensus.cc:1261] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 3 FOLLOWER]: Refusing update from remote peer 5fbb448689c949d98dec8e37cf5c3478: Log matching property violated. Preceding OpId in replica: term: 2 index: 313. Preceding OpId from leader: term: 3 index: 314. (index mismatch)
I20240502 14:00:20.620182 15206 consensus_queue.cc:1035] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7553d1bab6f642ac8395c928da1c58d7" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 34019 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s
I20240502 14:00:20.621966 13172 consensus_queue.cc:237] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 314, Committed index: 314, Last appended: 3.314, Last appended by leader: 313, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:20.623297 14723 raft_consensus.cc:1261] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 3 FOLLOWER]: Refusing update from remote peer 5fbb448689c949d98dec8e37cf5c3478: Log matching property violated. Preceding OpId in replica: term: 3 index: 314. Preceding OpId from leader: term: 3 index: 315. (index mismatch)
I20240502 14:00:20.623656 15206 consensus_queue.cc:1035] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 315, Last known committed idx: 314, Time since last communication: 0.000s
I20240502 14:00:20.624998 15152 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 [term 3 LEADER]: Committing config change with OpId 3.315: config changed from index 313 to 315, VOTER 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) evicted. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:20.625416 14723 raft_consensus.cc:2932] T b9bc1c84cb144965a21bdc40f099416d P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 3 FOLLOWER]: Committing config change with OpId 3.315: config changed from index 313 to 315, VOTER 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) evicted. New config: { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:20.626469 11612 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b9bc1c84cb144965a21bdc40f099416d with cas_config_opid_index 313: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:20.627238 11620 catalog_manager.cc:5484] T b9bc1c84cb144965a21bdc40f099416d P 5fbb448689c949d98dec8e37cf5c3478 reported cstate change: config changed from index 313 to 315, VOTER 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135) evicted. New cstate: current_term: 3 leader_uuid: "5fbb448689c949d98dec8e37cf5c3478" committed_config { opid_index: 315 OBSOLETE_local: false peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:20.629703 13414 tablet_service.cc:1514] Processing DeleteTablet for tablet b9bc1c84cb144965a21bdc40f099416d with delete_type TABLET_DATA_TOMBSTONED (TS 7553d1bab6f642ac8395c928da1c58d7 not found in new config with opid_index 315) from {username='slave'} at 127.0.0.1:33024
I20240502 14:00:20.630409 15210 tablet_replica.cc:330] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7: stopping tablet replica
I20240502 14:00:20.630746 15210 raft_consensus.cc:2229] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:20.630915 15210 raft_consensus.cc:2258] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:20.632462 15210 ts_tablet_manager.cc:1875] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:20.634637 15210 ts_tablet_manager.cc:1888] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.314
I20240502 14:00:20.634756 15210 log.cc:1196] T b9bc1c84cb144965a21bdc40f099416d P 7553d1bab6f642ac8395c928da1c58d7: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal/wals/b9bc1c84cb144965a21bdc40f099416d
I20240502 14:00:20.635339 11613 catalog_manager.cc:4830] TS 7553d1bab6f642ac8395c928da1c58d7 (127.8.86.135:34019): tablet b9bc1c84cb144965a21bdc40f099416d (table rebalance_test_table_4 [id=83b2409333254a509079ef88e0c156f9]) successfully deleted
I20240502 14:00:20.683722 13037 raft_consensus.cc:1261] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 2 FOLLOWER]: Refusing update from remote peer 1ca55a1c9fcb4a1487dcde45aec09e3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 322. Preceding OpId from leader: term: 2 index: 323. (index mismatch)
I20240502 14:00:20.684278 14998 consensus_queue.cc:1035] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 323, Last known committed idx: 322, Time since last communication: 0.000s
I20240502 14:00:20.687248 12026 raft_consensus.cc:1261] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 2 FOLLOWER]: Refusing update from remote peer 1ca55a1c9fcb4a1487dcde45aec09e3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 322. Preceding OpId from leader: term: 2 index: 323. (index mismatch)
I20240502 14:00:20.687669 15246 consensus_queue.cc:1035] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a078666d30488f9c204d93e6394ef9" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 45325 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 323, Last known committed idx: 322, Time since last communication: 0.000s
I20240502 14:00:20.692786 14723 consensus_queue.cc:237] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 323, Committed index: 323, Last appended: 2.323, Last appended by leader: 322, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:20.693742 13037 raft_consensus.cc:1261] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 2 FOLLOWER]: Refusing update from remote peer 1ca55a1c9fcb4a1487dcde45aec09e3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 323. Preceding OpId from leader: term: 2 index: 324. (index mismatch)
I20240502 14:00:20.694110 15246 consensus_queue.cc:1035] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 324, Last known committed idx: 323, Time since last communication: 0.000s
I20240502 14:00:20.695592 15247 raft_consensus.cc:2932] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 LEADER]: Committing config change with OpId 2.324: config changed from index 322 to 324, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:20.696027 13037 raft_consensus.cc:2932] T 1870fb0c459f49cab36812d16e30ddf5 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 2 FOLLOWER]: Committing config change with OpId 2.324: config changed from index 322 to 324, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:20.697325 11611 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1870fb0c459f49cab36812d16e30ddf5 with cas_config_opid_index 322: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:20.697850 11620 catalog_manager.cc:5484] T 1870fb0c459f49cab36812d16e30ddf5 P 1ca55a1c9fcb4a1487dcde45aec09e3c reported cstate change: config changed from index 322 to 324, VOTER 65a078666d30488f9c204d93e6394ef9 (127.8.86.131) evicted. New cstate: current_term: 2 leader_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" committed_config { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:20.700165 12008 tablet_service.cc:1514] Processing DeleteTablet for tablet 1870fb0c459f49cab36812d16e30ddf5 with delete_type TABLET_DATA_TOMBSTONED (TS 65a078666d30488f9c204d93e6394ef9 not found in new config with opid_index 324) from {username='slave'} at 127.0.0.1:38592
I20240502 14:00:20.700527 15250 tablet_replica.cc:330] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9: stopping tablet replica
I20240502 14:00:20.700654 15250 raft_consensus.cc:2229] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:20.700783 15250 raft_consensus.cc:2258] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:20.702101 15250 ts_tablet_manager.cc:1875] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:20.704043 15250 ts_tablet_manager.cc:1888] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.323
I20240502 14:00:20.704116 15250 log.cc:1196] T 1870fb0c459f49cab36812d16e30ddf5 P 65a078666d30488f9c204d93e6394ef9: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/wals/1870fb0c459f49cab36812d16e30ddf5
I20240502 14:00:20.704653 11612 catalog_manager.cc:4830] TS 65a078666d30488f9c204d93e6394ef9 (127.8.86.131:45325): tablet 1870fb0c459f49cab36812d16e30ddf5 (table rebalance_test_table_3 [id=22735e7421b84776ac7e9ae79a0aa1f2]) successfully deleted
I20240502 14:00:20.874505 13037 consensus_queue.cc:237] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 305, Committed index: 305, Last appended: 2.305, Last appended by leader: 304, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:20.875624 14723 raft_consensus.cc:1261] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 FOLLOWER]: Refusing update from remote peer 4f6f2d2b1569415cbbbea2466dc28e28: Log matching property violated. Preceding OpId in replica: term: 2 index: 305. Preceding OpId from leader: term: 2 index: 306. (index mismatch)
I20240502 14:00:20.876117 15251 consensus_queue.cc:1035] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 306, Last known committed idx: 305, Time since last communication: 0.000s
I20240502 14:00:20.877930 15150 raft_consensus.cc:2932] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 [term 2 LEADER]: Committing config change with OpId 2.306: config changed from index 304 to 306, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:20.879359 11613 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 11f585c28308464bb46aabdf0ad98a08 with cas_config_opid_index 304: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:20.880285 11617 catalog_manager.cc:5484] T 11f585c28308464bb46aabdf0ad98a08 P 4f6f2d2b1569415cbbbea2466dc28e28 reported cstate change: config changed from index 304 to 306, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New cstate: current_term: 2 leader_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" committed_config { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:20.881081 14723 raft_consensus.cc:2932] T 11f585c28308464bb46aabdf0ad98a08 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 2 FOLLOWER]: Committing config change with OpId 2.306: config changed from index 304 to 306, VOTER 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129) evicted. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "4f6f2d2b1569415cbbbea2466dc28e28" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 42709 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:20.883276 11747 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f585c28308464bb46aabdf0ad98a08 with delete_type TABLET_DATA_TOMBSTONED (TS 73ce3d001cdf4d0c8e1487958b7694ea not found in new config with opid_index 306) from {username='slave'} at 127.0.0.1:43190
I20240502 14:00:20.884821 15257 tablet_replica.cc:330] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea: stopping tablet replica
I20240502 14:00:20.884985 15257 raft_consensus.cc:2229] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:20.885123 15257 raft_consensus.cc:2258] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:20.886478 15257 ts_tablet_manager.cc:1875] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:20.888316 15257 ts_tablet_manager.cc:1888] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.305
I20240502 14:00:20.888391 15257 log.cc:1196] T 11f585c28308464bb46aabdf0ad98a08 P 73ce3d001cdf4d0c8e1487958b7694ea: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/wals/11f585c28308464bb46aabdf0ad98a08
I20240502 14:00:20.888964 11612 catalog_manager.cc:4830] TS 73ce3d001cdf4d0c8e1487958b7694ea (127.8.86.129:39411): tablet 11f585c28308464bb46aabdf0ad98a08 (table rebalance_test_table_2 [id=d77f8a4d2b7c4f42901744c7061e0d56]) successfully deleted
I20240502 14:00:20.945472 14723 raft_consensus.cc:1261] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 3 FOLLOWER]: Refusing update from remote peer 5fbb448689c949d98dec8e37cf5c3478: Log matching property violated. Preceding OpId in replica: term: 2 index: 304. Preceding OpId from leader: term: 3 index: 305. (index mismatch)
I20240502 14:00:20.946040 15206 consensus_queue.cc:1035] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 304, Time since last communication: 0.000s
I20240502 14:00:20.948968 11899 raft_consensus.cc:1261] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 3 FOLLOWER]: Refusing update from remote peer 5fbb448689c949d98dec8e37cf5c3478: Log matching property violated. Preceding OpId in replica: term: 2 index: 304. Preceding OpId from leader: term: 3 index: 305. (index mismatch)
I20240502 14:00:20.949513 15206 consensus_queue.cc:1035] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbf92f56d7144a22a6d482c73c7c7ea9" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 44377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 305, Last known committed idx: 304, Time since last communication: 0.000s
I20240502 14:00:20.950456 13172 consensus_queue.cc:237] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 305, Committed index: 305, Last appended: 3.305, Last appended by leader: 304, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } }
I20240502 14:00:20.951515 14723 raft_consensus.cc:1261] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 3 FOLLOWER]: Refusing update from remote peer 5fbb448689c949d98dec8e37cf5c3478: Log matching property violated. Preceding OpId in replica: term: 3 index: 305. Preceding OpId from leader: term: 3 index: 306. (index mismatch)
I20240502 14:00:20.951866 15206 consensus_queue.cc:1035] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 306, Last known committed idx: 305, Time since last communication: 0.000s
I20240502 14:00:20.953305 15152 raft_consensus.cc:2932] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 [term 3 LEADER]: Committing config change with OpId 3.306: config changed from index 304 to 306, VOTER cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) evicted. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:20.953738 14723 raft_consensus.cc:2932] T 3a50cea2712d48ca812cea7983d16840 P 1ca55a1c9fcb4a1487dcde45aec09e3c [term 3 FOLLOWER]: Committing config change with OpId 3.306: config changed from index 304 to 306, VOTER cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) evicted. New config: { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } } }
I20240502 14:00:20.954669 11612 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3a50cea2712d48ca812cea7983d16840 with cas_config_opid_index 304: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:20.955796 11617 catalog_manager.cc:5484] T 3a50cea2712d48ca812cea7983d16840 P 5fbb448689c949d98dec8e37cf5c3478 reported cstate change: config changed from index 304 to 306, VOTER cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130) evicted. New cstate: current_term: 3 leader_uuid: "5fbb448689c949d98dec8e37cf5c3478" committed_config { opid_index: 306 OBSOLETE_local: false peers { permanent_uuid: "5fbb448689c949d98dec8e37cf5c3478" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 44919 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ca55a1c9fcb4a1487dcde45aec09e3c" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 36081 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:20.958140 11877 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a50cea2712d48ca812cea7983d16840 with delete_type TABLET_DATA_TOMBSTONED (TS cbf92f56d7144a22a6d482c73c7c7ea9 not found in new config with opid_index 306) from {username='slave'} at 127.0.0.1:37684
I20240502 14:00:20.958562 15276 tablet_replica.cc:330] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9: stopping tablet replica
I20240502 14:00:20.958735 15276 raft_consensus.cc:2229] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:20.958856 15276 raft_consensus.cc:2258] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:20.960234 15276 ts_tablet_manager.cc:1875] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:20.962229 15276 ts_tablet_manager.cc:1888] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.305
I20240502 14:00:20.962327 15276 log.cc:1196] T 3a50cea2712d48ca812cea7983d16840 P cbf92f56d7144a22a6d482c73c7c7ea9: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/wals/3a50cea2712d48ca812cea7983d16840
I20240502 14:00:20.962886 11613 catalog_manager.cc:4830] TS cbf92f56d7144a22a6d482c73c7c7ea9 (127.8.86.130:44377): tablet 3a50cea2712d48ca812cea7983d16840 (table rebalance_test_table_1 [id=d6a2f7f1e4dc45ffbe7f9f2fba15577e]) successfully deleted
I20240502 14:00:21.169948 13151 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:21.171262 14704 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:21.172516 12008 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:21.183881 11747 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:21.184551 13020 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:21.184827 13414 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:21.195369 13283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:21.204223 11877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:21.291343 13151 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:21.293792 12008 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:21.294728 14704 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:21.298802 13020 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:21.302613 13414 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:21.303048 11747 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:21.327374 13283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:21.328338 11877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:21.414824 14704 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:21.417822 12008 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:21.419083 11747 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:21.419149 13283 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:21.423519 11877 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:21.430938 13414 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:21.436368 13020 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:21.456292 13151 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
12453ce20bf34d93b25ce6a40cfe613c | 127.8.86.190:44201 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.8.86.148:40301 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
1ca55a1c9fcb4a1487dcde45aec09e3c | 127.8.86.136:36081 | HEALTHY | <none> | 1 | 0
4f6f2d2b1569415cbbbea2466dc28e28 | 127.8.86.132:42709 | HEALTHY | <none> | 1 | 0
5fbb448689c949d98dec8e37cf5c3478 | 127.8.86.133:44919 | HEALTHY | <none> | 3 | 0
65a078666d30488f9c204d93e6394ef9 | 127.8.86.131:45325 | HEALTHY | <none> | 2 | 0
73ce3d001cdf4d0c8e1487958b7694ea | 127.8.86.129:39411 | HEALTHY | <none> | 3 | 0
7553d1bab6f642ac8395c928da1c58d7 | 127.8.86.135:34019 | HEALTHY | <none> | 0 | 0
a36d5a88b1a34006bd3d040f5a715766 | 127.8.86.134:36541 | HEALTHY | <none> | 2 | 0
cbf92f56d7144a22a6d482c73c7c7ea9 | 127.8.86.130:44377 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.8.86.129 | experimental | 127.8.86.129:39411
local_ip_for_outbound_sockets | 127.8.86.130 | experimental | 127.8.86.130:44377
local_ip_for_outbound_sockets | 127.8.86.131 | experimental | 127.8.86.131:45325
local_ip_for_outbound_sockets | 127.8.86.132 | experimental | 127.8.86.132:42709
local_ip_for_outbound_sockets | 127.8.86.133 | experimental | 127.8.86.133:44919
local_ip_for_outbound_sockets | 127.8.86.134 | experimental | 127.8.86.134:36541
local_ip_for_outbound_sockets | 127.8.86.135 | experimental | 127.8.86.135:34019
local_ip_for_outbound_sockets | 127.8.86.136 | experimental | 127.8.86.136:36081
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.8.86.129:39411
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.8.86.130:44377
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.8.86.131:45325
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.8.86.132:42709
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.8.86.133:44919
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.8.86.134:36541
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.8.86.135:34019
server_dump_info_path | /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.8.86.136:36081
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.8.86.148:40301 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 14:00:21.479470 8538 log_verifier.cc:126] Checking tablet 03409b5f413f499c919f2c6a5a4126f7
I20240502 14:00:21.490595 8538 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 03409b5f413f499c919f2c6a5a4126f7
I20240502 14:00:21.490651 8538 log_verifier.cc:126] Checking tablet 11f585c28308464bb46aabdf0ad98a08
I20240502 14:00:21.546267 8538 log_verifier.cc:177] Verified matching terms for 306 ops in tablet 11f585c28308464bb46aabdf0ad98a08
I20240502 14:00:21.546423 8538 log_verifier.cc:126] Checking tablet 182302167a0541888cb99bcd66635125
I20240502 14:00:21.602169 8538 log_verifier.cc:177] Verified matching terms for 338 ops in tablet 182302167a0541888cb99bcd66635125
I20240502 14:00:21.602311 8538 log_verifier.cc:126] Checking tablet 1870fb0c459f49cab36812d16e30ddf5
I20240502 14:00:21.661940 8538 log_verifier.cc:177] Verified matching terms for 324 ops in tablet 1870fb0c459f49cab36812d16e30ddf5
I20240502 14:00:21.662087 8538 log_verifier.cc:126] Checking tablet 236933482b6b41cf9e84a2eeb3b57638
I20240502 14:00:21.718530 8538 log_verifier.cc:177] Verified matching terms for 330 ops in tablet 236933482b6b41cf9e84a2eeb3b57638
I20240502 14:00:21.718664 8538 log_verifier.cc:126] Checking tablet 259e249430fa41a2bc2528b65d022b67
I20240502 14:00:21.769688 8538 log_verifier.cc:177] Verified matching terms for 299 ops in tablet 259e249430fa41a2bc2528b65d022b67
I20240502 14:00:21.769809 8538 log_verifier.cc:126] Checking tablet 3a50cea2712d48ca812cea7983d16840
I20240502 14:00:21.825356 8538 log_verifier.cc:177] Verified matching terms for 306 ops in tablet 3a50cea2712d48ca812cea7983d16840
I20240502 14:00:21.825505 8538 log_verifier.cc:126] Checking tablet 49041719e0ae42a390452b8716cc3ca7
I20240502 14:00:21.884464 8538 log_verifier.cc:177] Verified matching terms for 323 ops in tablet 49041719e0ae42a390452b8716cc3ca7
I20240502 14:00:21.884589 8538 log_verifier.cc:126] Checking tablet 7cd1b1872364411e86b810420c54034a
I20240502 14:00:21.934451 8538 log_verifier.cc:177] Verified matching terms for 277 ops in tablet 7cd1b1872364411e86b810420c54034a
I20240502 14:00:21.934585 8538 log_verifier.cc:126] Checking tablet 986b926ead014774925e3b8a3892c56d
I20240502 14:00:21.984259 8538 log_verifier.cc:177] Verified matching terms for 286 ops in tablet 986b926ead014774925e3b8a3892c56d
I20240502 14:00:21.984385 8538 log_verifier.cc:126] Checking tablet b9a6f7fca259430dac2ca17a8b180b2a
I20240502 14:00:22.037257 8538 log_verifier.cc:177] Verified matching terms for 315 ops in tablet b9a6f7fca259430dac2ca17a8b180b2a
I20240502 14:00:22.037412 8538 log_verifier.cc:126] Checking tablet b9bc1c84cb144965a21bdc40f099416d
I20240502 14:00:22.092590 8538 log_verifier.cc:177] Verified matching terms for 315 ops in tablet b9bc1c84cb144965a21bdc40f099416d
I20240502 14:00:22.092718 8538 log_verifier.cc:126] Checking tablet c0230720456b49339caaa9d313b6ce64
I20240502 14:00:22.139767 8538 log_verifier.cc:177] Verified matching terms for 261 ops in tablet c0230720456b49339caaa9d313b6ce64
I20240502 14:00:22.139887 8538 log_verifier.cc:126] Checking tablet c951304cdaa54040a81e51d076fdeaaf
I20240502 14:00:22.191299 8538 log_verifier.cc:177] Verified matching terms for 298 ops in tablet c951304cdaa54040a81e51d076fdeaaf
I20240502 14:00:22.191433 8538 log_verifier.cc:126] Checking tablet db4ce5e277504649a781e23861baa667
I20240502 14:00:22.250674 8538 log_verifier.cc:177] Verified matching terms for 330 ops in tablet db4ce5e277504649a781e23861baa667
I20240502 14:00:22.250813 8538 log_verifier.cc:126] Checking tablet dc89c51bfacd43d7a7cca53bc5624438
I20240502 14:00:22.314491 8538 log_verifier.cc:177] Verified matching terms for 377 ops in tablet dc89c51bfacd43d7a7cca53bc5624438
I20240502 14:00:22.316112 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 11686
I20240502 14:00:22.324630 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 11817
I20240502 14:00:22.332207 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 11948
I20240502 14:00:22.339354 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 12958
I20240502 14:00:22.345826 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 13089
I20240502 14:00:22.352236 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 13221
I20240502 14:00:22.358319 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 13352
I20240502 14:00:22.364535 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 14642
I20240502 14:00:22.370556 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 11595
2024-05-02T14:00:22Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/3 (27109 ms)
[----------] 1 test from RebalanceParamTest (27109 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20240502 14:00:22.426244 8538 test_util.cc:268] Using random seed: 103680637
I20240502 14:00:22.427070 8538 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:00:22.427107 8538 ts_itest-base.cc:113] --------------
I20240502 14:00:22.427126 8538 ts_itest-base.cc:114] 8 tablet servers
I20240502 14:00:22.427142 8538 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:00:22.427157 8538 ts_itest-base.cc:116] --------------
2024-05-02T14:00:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:00:22Z Disabled control of system clock
I20240502 14:00:22.435909 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.86.190:35811
--webserver_interface=127.8.86.190
--webserver_port=0
--builtin_ntp_servers=127.8.86.148:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.86.190:35811
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 14:00:22.544507 15398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:22.544746 15398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:22.544786 15398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:22.547811 15398 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:00:22.547884 15398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:00:22.547907 15398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:22.547925 15398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:00:22.547948 15398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:00:22.551695 15398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.8.86.190:35811
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.86.190:35811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.8.86.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.15398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:22.552608 15398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:22.553750 15398 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:22.561106 15403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:22.561702 15404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:22.562278 15406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:22.563095 15398 server_base.cc:1034] running on GCE node
I20240502 14:00:22.563607 15398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:22.564561 15398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:22.565721 15398 hybrid_clock.cc:648] HybridClock initialized: now 1714658422565700 us; error 42 us; skew 500 ppm
I20240502 14:00:22.567667 15398 webserver.cc:460] Webserver started at http://127.8.86.190:40335/ using document root <none> and password file <none>
I20240502 14:00:22.568228 15398 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:22.568295 15398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:22.568466 15398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:22.570106 15398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d9f1f4817ca9494da13bfe388be84b9d"
format_stamp: "Formatted at 2024-05-02 14:00:22 on dist-test-slave-h6v0"
I20240502 14:00:22.570644 15398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d9f1f4817ca9494da13bfe388be84b9d"
format_stamp: "Formatted at 2024-05-02 14:00:22 on dist-test-slave-h6v0"
I20240502 14:00:22.573967 15398 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240502 14:00:22.576197 15412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:22.577178 15398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:22.577293 15398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d9f1f4817ca9494da13bfe388be84b9d"
format_stamp: "Formatted at 2024-05-02 14:00:22 on dist-test-slave-h6v0"
I20240502 14:00:22.577384 15398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:22.591851 15398 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:22.592478 15398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:22.592639 15398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:22.599195 15398 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.190:35811
I20240502 14:00:22.599205 15464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.190:35811 every 8 connection(s)
I20240502 14:00:22.600118 15398 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:00:22.601122 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 15398
I20240502 14:00:22.601246 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:00:22.602787 15465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:22.607650 15465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d: Bootstrap starting.
I20240502 14:00:22.609881 15465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:22.610646 15465 log.cc:824] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:22.612179 15465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d: No bootstrap required, opened a new log
I20240502 14:00:22.614789 15465 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f1f4817ca9494da13bfe388be84b9d" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 35811 } }
I20240502 14:00:22.614965 15465 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:22.615016 15465 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f1f4817ca9494da13bfe388be84b9d, State: Initialized, Role: FOLLOWER
I20240502 14:00:22.615394 15465 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f1f4817ca9494da13bfe388be84b9d" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 35811 } }
I20240502 14:00:22.615517 15465 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:00:22.615579 15465 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:00:22.615690 15465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:22.616430 15465 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f1f4817ca9494da13bfe388be84b9d" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 35811 } }
I20240502 14:00:22.616750 15465 leader_election.cc:302] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9f1f4817ca9494da13bfe388be84b9d; no voters:
I20240502 14:00:22.617007 15465 leader_election.cc:288] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:00:22.617113 15470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:22.617609 15470 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d [term 1 LEADER]: Becoming Leader. State: Replica: d9f1f4817ca9494da13bfe388be84b9d, State: Running, Role: LEADER
I20240502 14:00:22.617888 15470 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f1f4817ca9494da13bfe388be84b9d" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 35811 } }
I20240502 14:00:22.618088 15465 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:00:22.619473 15472 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9f1f4817ca9494da13bfe388be84b9d. Latest consensus state: current_term: 1 leader_uuid: "d9f1f4817ca9494da13bfe388be84b9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f1f4817ca9494da13bfe388be84b9d" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 35811 } } }
I20240502 14:00:22.619506 15471 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9f1f4817ca9494da13bfe388be84b9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f1f4817ca9494da13bfe388be84b9d" member_type: VOTER last_known_addr { host: "127.8.86.190" port: 35811 } } }
I20240502 14:00:22.619576 15472 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d [sys.catalog]: This master's current role is: LEADER
I20240502 14:00:22.619599 15471 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d [sys.catalog]: This master's current role is: LEADER
I20240502 14:00:22.619957 15479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:00:22.622684 15479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:00:22.628564 15479 catalog_manager.cc:1347] Generated new cluster ID: b14324dba29e4fbdb862c0b82f0c70ae
I20240502 14:00:22.628991 15479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:00:22.633669 15479 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:00:22.634601 15479 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:00:22.644455 15479 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d9f1f4817ca9494da13bfe388be84b9d: Generated new TSK 0
I20240502 14:00:22.645323 15479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:00:22.655225 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.129:0
--local_ip_for_outbound_sockets=127.8.86.129
--webserver_interface=127.8.86.129
--webserver_port=0
--tserver_master_addrs=127.8.86.190:35811
--builtin_ntp_servers=127.8.86.148:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:00:22.759462 15489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:22.759677 15489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:22.759749 15489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:22.762667 15489 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:00:22.762733 15489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:22.762821 15489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.129
I20240502 14:00:22.766254 15489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.8.86.129
--webserver_port=0
--tserver_master_addrs=127.8.86.190:35811
--never_fsync=true
--heap_profile_path=/tmp/kudu.15489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.129
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:22.767071 15489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:22.768163 15489 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:22.779489 15497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:22.779886 15494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:22.780562 15495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:22.781790 15489 server_base.cc:1034] running on GCE node
I20240502 14:00:22.782235 15489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:22.782908 15489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:22.784083 15489 hybrid_clock.cc:648] HybridClock initialized: now 1714658422784066 us; error 35 us; skew 500 ppm
I20240502 14:00:22.786477 15489 webserver.cc:460] Webserver started at http://127.8.86.129:44755/ using document root <none> and password file <none>
I20240502 14:00:22.787173 15489 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:22.787258 15489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:22.787487 15489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:22.789738 15489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
format_stamp: "Formatted at 2024-05-02 14:00:22 on dist-test-slave-h6v0"
I20240502 14:00:22.790340 15489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
format_stamp: "Formatted at 2024-05-02 14:00:22 on dist-test-slave-h6v0"
I20240502 14:00:22.794512 15489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:00:22.797304 15503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:22.798605 15489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:00:22.798729 15489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
format_stamp: "Formatted at 2024-05-02 14:00:22 on dist-test-slave-h6v0"
I20240502 14:00:22.798835 15489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:22.829591 15489 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:22.830145 15489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:22.830307 15489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:22.830796 15489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:22.831594 15489 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:22.831640 15489 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:22.831676 15489 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:22.831696 15489 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:22.839413 15489 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.129:36237
I20240502 14:00:22.839473 15616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.129:36237 every 8 connection(s)
I20240502 14:00:22.841276 15489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:00:22.849510 15617 heartbeater.cc:347] Connected to a master server at 127.8.86.190:35811
I20240502 14:00:22.849718 15617 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:22.850199 15617 heartbeater.cc:511] Master 127.8.86.190:35811 requested a full tablet report, sending...
I20240502 14:00:22.851405 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 15489
I20240502 14:00:22.851507 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:00:22.851645 15429 ts_manager.cc:194] Registered new tserver with Master: 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:22.852788 15429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.129:34627
I20240502 14:00:22.853653 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.130:0
--local_ip_for_outbound_sockets=127.8.86.130
--webserver_interface=127.8.86.130
--webserver_port=0
--tserver_master_addrs=127.8.86.190:35811
--builtin_ntp_servers=127.8.86.148:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:00:22.958567 15620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:22.958775 15620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:22.958827 15620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:22.961822 15620 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:00:22.961889 15620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:22.962014 15620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.130
I20240502 14:00:22.965602 15620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.8.86.130
--webserver_port=0
--tserver_master_addrs=127.8.86.190:35811
--never_fsync=true
--heap_profile_path=/tmp/kudu.15620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.130
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:22.966486 15620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:22.967586 15620 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:22.975903 15628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:22.976226 15625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:22.976905 15626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:22.977699 15620 server_base.cc:1034] running on GCE node
I20240502 14:00:22.978032 15620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:22.978610 15620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:22.979760 15620 hybrid_clock.cc:648] HybridClock initialized: now 1714658422979752 us; error 32 us; skew 500 ppm
I20240502 14:00:22.981637 15620 webserver.cc:460] Webserver started at http://127.8.86.130:39697/ using document root <none> and password file <none>
I20240502 14:00:22.982187 15620 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:22.982251 15620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:22.982451 15620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:22.984105 15620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8484181c4ece483c8b408817fab1a917"
format_stamp: "Formatted at 2024-05-02 14:00:22 on dist-test-slave-h6v0"
I20240502 14:00:22.984587 15620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8484181c4ece483c8b408817fab1a917"
format_stamp: "Formatted at 2024-05-02 14:00:22 on dist-test-slave-h6v0"
I20240502 14:00:22.987987 15620 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:00:22.990252 15634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:22.991318 15620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:00:22.991438 15620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8484181c4ece483c8b408817fab1a917"
format_stamp: "Formatted at 2024-05-02 14:00:22 on dist-test-slave-h6v0"
I20240502 14:00:22.991544 15620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:23.003185 15620 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:23.003774 15620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:23.003945 15620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:23.004465 15620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:23.005314 15620 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:23.005367 15620 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:23.005427 15620 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:23.005455 15620 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:23.013239 15620 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.130:42261
I20240502 14:00:23.013288 15747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.130:42261 every 8 connection(s)
I20240502 14:00:23.014137 15620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:00:23.018980 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 15620
I20240502 14:00:23.019124 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:00:23.021384 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.131:0
--local_ip_for_outbound_sockets=127.8.86.131
--webserver_interface=127.8.86.131
--webserver_port=0
--tserver_master_addrs=127.8.86.190:35811
--builtin_ntp_servers=127.8.86.148:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:00:23.023254 15748 heartbeater.cc:347] Connected to a master server at 127.8.86.190:35811
I20240502 14:00:23.023515 15748 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:23.024080 15748 heartbeater.cc:511] Master 127.8.86.190:35811 requested a full tablet report, sending...
I20240502 14:00:23.025135 15429 ts_manager.cc:194] Registered new tserver with Master: 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:23.025979 15429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.130:58945
W20240502 14:00:23.127168 15751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:23.127377 15751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:23.127420 15751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:23.130337 15751 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:00:23.130406 15751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:23.130473 15751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.131
I20240502 14:00:23.133884 15751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.8.86.131
--webserver_port=0
--tserver_master_addrs=127.8.86.190:35811
--never_fsync=true
--heap_profile_path=/tmp/kudu.15751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.131
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:23.134742 15751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:23.135788 15751 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:23.143810 15757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:23.144709 15756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:23.145200 15759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:23.146095 15751 server_base.cc:1034] running on GCE node
I20240502 14:00:23.146517 15751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:23.147029 15751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:23.148187 15751 hybrid_clock.cc:648] HybridClock initialized: now 1714658423148165 us; error 35 us; skew 500 ppm
I20240502 14:00:23.150044 15751 webserver.cc:460] Webserver started at http://127.8.86.131:44951/ using document root <none> and password file <none>
I20240502 14:00:23.150606 15751 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:23.150668 15751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:23.150841 15751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:23.152781 15751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "82d7b7201cfc431d90baca1ad0a397cf"
format_stamp: "Formatted at 2024-05-02 14:00:23 on dist-test-slave-h6v0"
I20240502 14:00:23.153227 15751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "82d7b7201cfc431d90baca1ad0a397cf"
format_stamp: "Formatted at 2024-05-02 14:00:23 on dist-test-slave-h6v0"
I20240502 14:00:23.156378 15751 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:00:23.158411 15765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:23.159401 15751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:23.159507 15751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "82d7b7201cfc431d90baca1ad0a397cf"
format_stamp: "Formatted at 2024-05-02 14:00:23 on dist-test-slave-h6v0"
I20240502 14:00:23.159585 15751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:23.186192 15751 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:23.186869 15751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:23.187042 15751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:23.187585 15751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:23.188472 15751 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:23.188524 15751 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:23.188584 15751 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:23.188609 15751 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:23.197336 15751 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.131:41531
I20240502 14:00:23.197386 15878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.131:41531 every 8 connection(s)
I20240502 14:00:23.198256 15751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:00:23.206498 15879 heartbeater.cc:347] Connected to a master server at 127.8.86.190:35811
I20240502 14:00:23.206730 15879 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:23.207247 15879 heartbeater.cc:511] Master 127.8.86.190:35811 requested a full tablet report, sending...
I20240502 14:00:23.207582 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 15751
I20240502 14:00:23.207695 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:00:23.208266 15429 ts_manager.cc:194] Registered new tserver with Master: 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:23.208820 15429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.131:45049
I20240502 14:00:23.209787 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.132:0
--local_ip_for_outbound_sockets=127.8.86.132
--webserver_interface=127.8.86.132
--webserver_port=0
--tserver_master_addrs=127.8.86.190:35811
--builtin_ntp_servers=127.8.86.148:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:00:23.316419 15882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:23.316644 15882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:23.316686 15882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:23.320046 15882 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:00:23.320163 15882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:23.320245 15882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.132
I20240502 14:00:23.323876 15882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.8.86.132
--webserver_port=0
--tserver_master_addrs=127.8.86.190:35811
--never_fsync=true
--heap_profile_path=/tmp/kudu.15882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.132
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:23.324775 15882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:23.325917 15882 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:23.334113 15888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:23.334684 15887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:23.335177 15890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:23.336072 15882 server_base.cc:1034] running on GCE node
I20240502 14:00:23.336432 15882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:23.336967 15882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:23.338112 15882 hybrid_clock.cc:648] HybridClock initialized: now 1714658423338099 us; error 29 us; skew 500 ppm
I20240502 14:00:23.340013 15882 webserver.cc:460] Webserver started at http://127.8.86.132:38959/ using document root <none> and password file <none>
I20240502 14:00:23.340560 15882 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:23.340615 15882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:23.340777 15882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:23.342442 15882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
format_stamp: "Formatted at 2024-05-02 14:00:23 on dist-test-slave-h6v0"
I20240502 14:00:23.342916 15882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
format_stamp: "Formatted at 2024-05-02 14:00:23 on dist-test-slave-h6v0"
I20240502 14:00:23.346065 15882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:00:23.348237 15896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:23.349243 15882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:23.349349 15882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
format_stamp: "Formatted at 2024-05-02 14:00:23 on dist-test-slave-h6v0"
I20240502 14:00:23.349439 15882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:23.378320 15882 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:23.378891 15882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:23.379036 15882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:23.379524 15882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:23.380331 15882 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:23.380380 15882 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:23.380437 15882 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:23.380470 15882 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:23.388438 15882 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.132:45719
I20240502 14:00:23.388476 16009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.132:45719 every 8 connection(s)
I20240502 14:00:23.389287 15882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:00:23.395839 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 15882
I20240502 14:00:23.395961 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:00:23.397927 16010 heartbeater.cc:347] Connected to a master server at 127.8.86.190:35811
I20240502 14:00:23.398151 16010 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:23.398221 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.133:0
--local_ip_for_outbound_sockets=127.8.86.133
--webserver_interface=127.8.86.133
--webserver_port=0
--tserver_master_addrs=127.8.86.190:35811
--builtin_ntp_servers=127.8.86.148:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:00:23.398788 16010 heartbeater.cc:511] Master 127.8.86.190:35811 requested a full tablet report, sending...
I20240502 14:00:23.399852 15429 ts_manager.cc:194] Registered new tserver with Master: 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:23.400586 15429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.132:34959
W20240502 14:00:23.502701 16013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:23.502910 16013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:23.502957 16013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:23.506027 16013 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:00:23.506089 16013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:23.506175 16013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.133
I20240502 14:00:23.509696 16013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.86.133
--webserver_port=0
--tserver_master_addrs=127.8.86.190:35811
--never_fsync=true
--heap_profile_path=/tmp/kudu.16013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.133
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:23.510541 16013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:23.511615 16013 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:23.519762 16018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:23.520148 16021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:23.520752 16019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:23.521528 16013 server_base.cc:1034] running on GCE node
I20240502 14:00:23.521889 16013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:23.522440 16013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:23.523586 16013 hybrid_clock.cc:648] HybridClock initialized: now 1714658423523566 us; error 33 us; skew 500 ppm
I20240502 14:00:23.525426 16013 webserver.cc:460] Webserver started at http://127.8.86.133:38045/ using document root <none> and password file <none>
I20240502 14:00:23.525974 16013 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:23.526029 16013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:23.526209 16013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:23.527859 16013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "f688370715454d6e814a78fcf85ac5c3"
format_stamp: "Formatted at 2024-05-02 14:00:23 on dist-test-slave-h6v0"
I20240502 14:00:23.528323 16013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "f688370715454d6e814a78fcf85ac5c3"
format_stamp: "Formatted at 2024-05-02 14:00:23 on dist-test-slave-h6v0"
I20240502 14:00:23.531589 16013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:00:23.533602 16027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:23.534588 16013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:00:23.534680 16013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f688370715454d6e814a78fcf85ac5c3"
format_stamp: "Formatted at 2024-05-02 14:00:23 on dist-test-slave-h6v0"
I20240502 14:00:23.534752 16013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:23.549095 16013 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:23.549595 16013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:23.549719 16013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:23.550161 16013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:23.550997 16013 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:23.551041 16013 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:23.551079 16013 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:23.551098 16013 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:23.559608 16013 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.133:42151
I20240502 14:00:23.559655 16140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.133:42151 every 8 connection(s)
I20240502 14:00:23.560495 16013 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:00:23.564172 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 16013
I20240502 14:00:23.564285 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:00:23.566382 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.134:0
--local_ip_for_outbound_sockets=127.8.86.134
--webserver_interface=127.8.86.134
--webserver_port=0
--tserver_master_addrs=127.8.86.190:35811
--builtin_ntp_servers=127.8.86.148:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:00:23.569162 16141 heartbeater.cc:347] Connected to a master server at 127.8.86.190:35811
I20240502 14:00:23.569372 16141 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:23.569872 16141 heartbeater.cc:511] Master 127.8.86.190:35811 requested a full tablet report, sending...
I20240502 14:00:23.570869 15429 ts_manager.cc:194] Registered new tserver with Master: f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
I20240502 14:00:23.571394 15429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.133:52297
W20240502 14:00:23.676288 16144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:23.676522 16144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:23.676596 16144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:23.679630 16144 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:00:23.679702 16144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:23.679795 16144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.134
I20240502 14:00:23.683403 16144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.8.86.134
--webserver_port=0
--tserver_master_addrs=127.8.86.190:35811
--never_fsync=true
--heap_profile_path=/tmp/kudu.16144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.134
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:23.684248 16144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:23.685323 16144 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:23.693797 16150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:23.694279 16149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:23.694785 16152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:23.695607 16144 server_base.cc:1034] running on GCE node
I20240502 14:00:23.695986 16144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:23.696529 16144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:23.697667 16144 hybrid_clock.cc:648] HybridClock initialized: now 1714658423697652 us; error 41 us; skew 500 ppm
I20240502 14:00:23.699483 16144 webserver.cc:460] Webserver started at http://127.8.86.134:33023/ using document root <none> and password file <none>
I20240502 14:00:23.700013 16144 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:23.700068 16144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:23.700237 16144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:23.701841 16144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "3c3658c946994e2ba9f41b64aa327d25"
format_stamp: "Formatted at 2024-05-02 14:00:23 on dist-test-slave-h6v0"
I20240502 14:00:23.702401 16144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "3c3658c946994e2ba9f41b64aa327d25"
format_stamp: "Formatted at 2024-05-02 14:00:23 on dist-test-slave-h6v0"
I20240502 14:00:23.705716 16144 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 14:00:23.707840 16158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:23.708781 16144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:23.708887 16144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "3c3658c946994e2ba9f41b64aa327d25"
format_stamp: "Formatted at 2024-05-02 14:00:23 on dist-test-slave-h6v0"
I20240502 14:00:23.708974 16144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:23.733860 16144 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:23.734470 16144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:23.734622 16144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:23.735124 16144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:23.735980 16144 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:23.736037 16144 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:23.736094 16144 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:23.736122 16144 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:23.744287 16144 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.134:34379
I20240502 14:00:23.744335 16271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.134:34379 every 8 connection(s)
I20240502 14:00:23.745162 16144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:00:23.752370 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 16144
I20240502 14:00:23.752532 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 14:00:23.753463 16272 heartbeater.cc:347] Connected to a master server at 127.8.86.190:35811
I20240502 14:00:23.753669 16272 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:23.754182 16272 heartbeater.cc:511] Master 127.8.86.190:35811 requested a full tablet report, sending...
I20240502 14:00:23.754850 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.135:0
--local_ip_for_outbound_sockets=127.8.86.135
--webserver_interface=127.8.86.135
--webserver_port=0
--tserver_master_addrs=127.8.86.190:35811
--builtin_ntp_servers=127.8.86.148:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:00:23.755198 15429 ts_manager.cc:194] Registered new tserver with Master: 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379)
I20240502 14:00:23.755726 15429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.134:60897
I20240502 14:00:23.854959 15617 heartbeater.cc:503] Master 127.8.86.190:35811 was elected leader, sending a full tablet report...
W20240502 14:00:23.859078 16275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:23.859261 16275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:23.859311 16275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:23.862205 16275 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:00:23.862257 16275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:23.862375 16275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.135
I20240502 14:00:23.865904 16275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.8.86.135
--webserver_port=0
--tserver_master_addrs=127.8.86.190:35811
--never_fsync=true
--heap_profile_path=/tmp/kudu.16275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.135
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:23.866706 16275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:23.867703 16275 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:23.875491 16280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:23.875922 16281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:23.876533 16283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:23.877288 16275 server_base.cc:1034] running on GCE node
I20240502 14:00:23.877609 16275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:23.878082 16275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:23.879240 16275 hybrid_clock.cc:648] HybridClock initialized: now 1714658423879215 us; error 39 us; skew 500 ppm
I20240502 14:00:23.880931 16275 webserver.cc:460] Webserver started at http://127.8.86.135:35089/ using document root <none> and password file <none>
I20240502 14:00:23.881422 16275 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:23.881501 16275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:23.881668 16275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:23.883212 16275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "8fea283177394bd28d7eacf4fb744d59"
format_stamp: "Formatted at 2024-05-02 14:00:23 on dist-test-slave-h6v0"
I20240502 14:00:23.883644 16275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "8fea283177394bd28d7eacf4fb744d59"
format_stamp: "Formatted at 2024-05-02 14:00:23 on dist-test-slave-h6v0"
I20240502 14:00:23.886706 16275 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:00:23.888628 16289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:23.889545 16275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:23.889658 16275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "8fea283177394bd28d7eacf4fb744d59"
format_stamp: "Formatted at 2024-05-02 14:00:23 on dist-test-slave-h6v0"
I20240502 14:00:23.889745 16275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:23.901664 16275 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:23.902159 16275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:23.902348 16275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:23.902813 16275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:23.903584 16275 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:23.903628 16275 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:23.903663 16275 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:23.903688 16275 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:23.911710 16275 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.135:46245
I20240502 14:00:23.911739 16402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.135:46245 every 8 connection(s)
I20240502 14:00:23.912541 16275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:00:23.920126 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 16275
I20240502 14:00:23.920245 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 14:00:23.920624 16403 heartbeater.cc:347] Connected to a master server at 127.8.86.190:35811
I20240502 14:00:23.920841 16403 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:23.921324 16403 heartbeater.cc:511] Master 127.8.86.190:35811 requested a full tablet report, sending...
I20240502 14:00:23.922240 15429 ts_manager.cc:194] Registered new tserver with Master: 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245)
I20240502 14:00:23.922487 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.136:0
--local_ip_for_outbound_sockets=127.8.86.136
--webserver_interface=127.8.86.136
--webserver_port=0
--tserver_master_addrs=127.8.86.190:35811
--builtin_ntp_servers=127.8.86.148:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:00:23.922991 15429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.135:37391
W20240502 14:00:24.023499 16406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:24.023697 16406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:24.023766 16406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:24.026726 16406 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:00:24.026788 16406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:24.026895 16406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.136
I20240502 14:00:24.028020 15748 heartbeater.cc:503] Master 127.8.86.190:35811 was elected leader, sending a full tablet report...
I20240502 14:00:24.030553 16406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.8.86.136
--webserver_port=0
--tserver_master_addrs=127.8.86.190:35811
--never_fsync=true
--heap_profile_path=/tmp/kudu.16406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.136
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:24.031246 16406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:24.032196 16406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:24.040030 16411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:24.040788 16412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:24.041275 16414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:24.042121 16406 server_base.cc:1034] running on GCE node
I20240502 14:00:24.042483 16406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:24.042968 16406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:24.044122 16406 hybrid_clock.cc:648] HybridClock initialized: now 1714658424044109 us; error 28 us; skew 500 ppm
I20240502 14:00:24.045820 16406 webserver.cc:460] Webserver started at http://127.8.86.136:40225/ using document root <none> and password file <none>
I20240502 14:00:24.046356 16406 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:24.046418 16406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:24.046590 16406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:24.048139 16406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "2254b3ee3db647598d781e467584f78d"
format_stamp: "Formatted at 2024-05-02 14:00:24 on dist-test-slave-h6v0"
I20240502 14:00:24.048568 16406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "2254b3ee3db647598d781e467584f78d"
format_stamp: "Formatted at 2024-05-02 14:00:24 on dist-test-slave-h6v0"
I20240502 14:00:24.051602 16406 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240502 14:00:24.053562 16420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:24.054509 16406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:24.054603 16406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "2254b3ee3db647598d781e467584f78d"
format_stamp: "Formatted at 2024-05-02 14:00:24 on dist-test-slave-h6v0"
I20240502 14:00:24.054682 16406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:24.067883 16406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:24.068348 16406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:24.068485 16406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:24.068985 16406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:24.069734 16406 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:24.069782 16406 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:24.069835 16406 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:24.069864 16406 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:24.078032 16406 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.136:38631
I20240502 14:00:24.078148 16533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.136:38631 every 8 connection(s)
I20240502 14:00:24.078910 16406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:00:24.087121 16534 heartbeater.cc:347] Connected to a master server at 127.8.86.190:35811
I20240502 14:00:24.087309 16534 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:24.087738 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 16406
I20240502 14:00:24.087769 16534 heartbeater.cc:511] Master 127.8.86.190:35811 requested a full tablet report, sending...
I20240502 14:00:24.087857 8538 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 14:00:24.088757 15429 ts_manager.cc:194] Registered new tserver with Master: 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631)
I20240502 14:00:24.089385 15429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.136:55019
I20240502 14:00:24.090166 8538 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 14:00:24.099299 8538 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 14:00:24.099361 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 16013
I20240502 14:00:24.105386 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 16144
I20240502 14:00:24.110666 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 16275
I20240502 14:00:24.115682 8538 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4LMC7s/build/debug/bin/kudu with pid 16406
I20240502 14:00:24.211050 15879 heartbeater.cc:503] Master 127.8.86.190:35811 was elected leader, sending a full tablet report...
I20240502 14:00:24.402472 16010 heartbeater.cc:503] Master 127.8.86.190:35811 was elected leader, sending a full tablet report...
I20240502 14:00:27.872324 15428 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49520:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:27.891801 15812 tablet_service.cc:1467] Processing CreateTablet for tablet d81f14b7c1d1468196d0fe6d67fe93a7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fcc590955678424ab1c27b0b23f75544]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:27.891801 15680 tablet_service.cc:1467] Processing CreateTablet for tablet b07f4db6f82e4a56accdbf1bbbc5f965 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fcc590955678424ab1c27b0b23f75544]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:27.891803 15682 tablet_service.cc:1467] Processing CreateTablet for tablet 9391f16ae2d247ec8a71e5c971cdda71 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fcc590955678424ab1c27b0b23f75544]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:27.892078 15943 tablet_service.cc:1467] Processing CreateTablet for tablet b07f4db6f82e4a56accdbf1bbbc5f965 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fcc590955678424ab1c27b0b23f75544]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:27.892786 15680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b07f4db6f82e4a56accdbf1bbbc5f965. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.892786 15812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81f14b7c1d1468196d0fe6d67fe93a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.892838 15943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b07f4db6f82e4a56accdbf1bbbc5f965. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.893087 15682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9391f16ae2d247ec8a71e5c971cdda71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.894471 15813 tablet_service.cc:1467] Processing CreateTablet for tablet 9391f16ae2d247ec8a71e5c971cdda71 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fcc590955678424ab1c27b0b23f75544]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:27.894721 15813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9391f16ae2d247ec8a71e5c971cdda71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.894800 15681 tablet_service.cc:1467] Processing CreateTablet for tablet d81f14b7c1d1468196d0fe6d67fe93a7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fcc590955678424ab1c27b0b23f75544]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:27.895033 15681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81f14b7c1d1468196d0fe6d67fe93a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.898736 15944 tablet_service.cc:1467] Processing CreateTablet for tablet 9391f16ae2d247ec8a71e5c971cdda71 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fcc590955678424ab1c27b0b23f75544]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:27.899005 15944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9391f16ae2d247ec8a71e5c971cdda71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.900310 16556 tablet_bootstrap.cc:492] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:27.901916 16555 tablet_bootstrap.cc:492] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:27.902384 16556 tablet_bootstrap.cc:654] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.902665 16554 tablet_bootstrap.cc:492] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:27.903179 16556 log.cc:824] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:27.903754 16555 tablet_bootstrap.cc:654] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.904466 16555 log.cc:824] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:27.904877 16556 tablet_bootstrap.cc:492] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:27.905043 16556 ts_tablet_manager.cc:1367] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240502 14:00:27.905750 16554 tablet_bootstrap.cc:654] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.906188 16555 tablet_bootstrap.cc:492] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:27.906374 16555 ts_tablet_manager.cc:1367] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:00:27.906657 16554 log.cc:824] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:27.906466 15551 tablet_service.cc:1467] Processing CreateTablet for tablet d81f14b7c1d1468196d0fe6d67fe93a7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fcc590955678424ab1c27b0b23f75544]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:27.907301 15551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81f14b7c1d1468196d0fe6d67fe93a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.908740 16556 raft_consensus.cc:348] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:27.908950 16556 raft_consensus.cc:374] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.909003 16556 raft_consensus.cc:729] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.909155 16555 raft_consensus.cc:348] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:27.909343 16555 raft_consensus.cc:374] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.909404 16555 raft_consensus.cc:729] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.909426 16556 consensus_queue.cc:260] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:27.909832 16555 consensus_queue.cc:260] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:27.910313 16556 ts_tablet_manager.cc:1398] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20240502 14:00:27.910554 16556 tablet_bootstrap.cc:492] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:27.906095 15550 tablet_service.cc:1467] Processing CreateTablet for tablet b07f4db6f82e4a56accdbf1bbbc5f965 (DEFAULT_TABLE table=rebalance_test_table_0 [id=fcc590955678424ab1c27b0b23f75544]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:27.910811 15550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b07f4db6f82e4a56accdbf1bbbc5f965. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:27.911443 16555 ts_tablet_manager.cc:1398] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:00:27.911473 16556 tablet_bootstrap.cc:654] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.911759 16555 tablet_bootstrap.cc:492] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:27.912622 16555 tablet_bootstrap.cc:654] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.915726 16555 tablet_bootstrap.cc:492] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:27.915788 16556 tablet_bootstrap.cc:492] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:27.915812 16555 ts_tablet_manager.cc:1367] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:27.915884 16556 ts_tablet_manager.cc:1367] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:27.916357 16556 raft_consensus.cc:348] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:27.916455 16556 raft_consensus.cc:374] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.916375 16555 raft_consensus.cc:348] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:27.916486 16556 raft_consensus.cc:729] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.916496 16555 raft_consensus.cc:374] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.916528 16555 raft_consensus.cc:729] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.916610 16556 consensus_queue.cc:260] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:27.916664 16555 consensus_queue.cc:260] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:27.916828 16554 tablet_bootstrap.cc:492] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:27.916883 16556 ts_tablet_manager.cc:1398] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:27.917016 16554 ts_tablet_manager.cc:1367] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.003s
I20240502 14:00:27.917061 16555 ts_tablet_manager.cc:1398] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:27.917230 16555 tablet_bootstrap.cc:492] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:27.914811 16564 tablet_bootstrap.cc:492] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:27.918182 16555 tablet_bootstrap.cc:654] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.919540 16555 tablet_bootstrap.cc:492] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:27.919636 16555 ts_tablet_manager.cc:1367] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:27.920135 16564 tablet_bootstrap.cc:654] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.920132 16555 raft_consensus.cc:348] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:27.920214 16555 raft_consensus.cc:374] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.920253 16555 raft_consensus.cc:729] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.920379 16555 consensus_queue.cc:260] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:27.920598 16555 ts_tablet_manager.cc:1398] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:27.920907 16564 log.cc:824] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:27.921505 16554 raft_consensus.cc:348] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:27.921692 16554 raft_consensus.cc:374] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.921747 16554 raft_consensus.cc:729] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.922187 16554 consensus_queue.cc:260] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:27.922646 16564 tablet_bootstrap.cc:492] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:27.922813 16564 ts_tablet_manager.cc:1367] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20240502 14:00:27.923084 16554 ts_tablet_manager.cc:1398] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20240502 14:00:27.923317 16554 tablet_bootstrap.cc:492] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:27.924204 16554 tablet_bootstrap.cc:654] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.925346 16554 tablet_bootstrap.cc:492] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:27.925433 16554 ts_tablet_manager.cc:1367] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:27.925942 16554 raft_consensus.cc:348] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:27.926072 16554 raft_consensus.cc:374] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.926116 16554 raft_consensus.cc:729] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.926347 16554 consensus_queue.cc:260] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:27.926609 16564 raft_consensus.cc:348] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:27.926754 16554 ts_tablet_manager.cc:1398] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:27.926805 16564 raft_consensus.cc:374] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.926863 16564 raft_consensus.cc:729] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.927196 16564 consensus_queue.cc:260] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:27.928063 16564 ts_tablet_manager.cc:1398] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20240502 14:00:27.928294 16564 tablet_bootstrap.cc:492] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:27.929132 16564 tablet_bootstrap.cc:654] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:27.930238 16564 tablet_bootstrap.cc:492] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:27.930348 16564 ts_tablet_manager.cc:1367] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:00:27.930913 16564 raft_consensus.cc:348] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:27.930997 16564 raft_consensus.cc:374] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:27.931030 16564 raft_consensus.cc:729] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:27.931149 16564 consensus_queue.cc:260] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:27.931358 16564 ts_tablet_manager.cc:1398] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 14:00:27.952670 15880 tablet.cc:2369] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:27.962427 16561 raft_consensus.cc:482] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:27.962638 16561 raft_consensus.cc:504] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:27.963752 16561 leader_election.cc:288] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237), 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:27.967763 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07f4db6f82e4a56accdbf1bbbc5f965" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" is_pre_election: true
I20240502 14:00:27.967769 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07f4db6f82e4a56accdbf1bbbc5f965" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" is_pre_election: true
I20240502 14:00:27.968052 15964 raft_consensus.cc:2454] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 0.
I20240502 14:00:27.968052 15571 raft_consensus.cc:2454] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 0.
I20240502 14:00:27.968444 15635 leader_election.cc:302] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:27.968683 16561 raft_consensus.cc:2781] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:27.968761 16561 raft_consensus.cc:482] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:27.968819 16561 raft_consensus.cc:3037] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:27.969683 16561 raft_consensus.cc:504] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:27.969996 16561 leader_election.cc:288] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 election: Requested vote from peers 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237), 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:27.970336 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07f4db6f82e4a56accdbf1bbbc5f965" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
I20240502 14:00:27.970336 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b07f4db6f82e4a56accdbf1bbbc5f965" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
I20240502 14:00:27.970453 15964 raft_consensus.cc:3037] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:27.970453 15571 raft_consensus.cc:3037] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:27.971472 15964 raft_consensus.cc:2454] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 1.
I20240502 14:00:27.971472 15571 raft_consensus.cc:2454] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 1.
I20240502 14:00:27.971745 15635 leader_election.cc:302] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:27.971909 16561 raft_consensus.cc:2781] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:27.972097 16561 raft_consensus.cc:686] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Becoming Leader. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Running, Role: LEADER
I20240502 14:00:27.972401 16561 consensus_queue.cc:237] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:27.975281 15429 catalog_manager.cc:5484] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8484181c4ece483c8b408817fab1a917 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } }
W20240502 14:00:28.018623 15749 tablet.cc:2369] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:00:28.096191 15618 tablet.cc:2369] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:00:28.143641 16011 tablet.cc:2369] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:28.226336 16560 raft_consensus.cc:482] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:28.226490 16560 raft_consensus.cc:504] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:28.227608 16560 leader_election.cc:288] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531), 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:28.231536 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9391f16ae2d247ec8a71e5c971cdda71" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917" is_pre_election: true
I20240502 14:00:28.231532 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9391f16ae2d247ec8a71e5c971cdda71" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf" is_pre_election: true
I20240502 14:00:28.231782 15702 raft_consensus.cc:2454] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 0.
I20240502 14:00:28.231837 15833 raft_consensus.cc:2454] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 0.
I20240502 14:00:28.232237 15897 leader_election.cc:302] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:28.232474 16560 raft_consensus.cc:2781] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:28.232543 16560 raft_consensus.cc:482] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:28.232594 16560 raft_consensus.cc:3037] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.233696 16560 raft_consensus.cc:504] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:28.234026 16560 leader_election.cc:288] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 election: Requested vote from peers 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531), 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:28.234392 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9391f16ae2d247ec8a71e5c971cdda71" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf"
I20240502 14:00:28.234437 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9391f16ae2d247ec8a71e5c971cdda71" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917"
I20240502 14:00:28.234529 15833 raft_consensus.cc:3037] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.234566 15702 raft_consensus.cc:3037] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.235396 15702 raft_consensus.cc:2454] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 1.
I20240502 14:00:28.235451 15833 raft_consensus.cc:2454] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 1.
I20240502 14:00:28.235716 15897 leader_election.cc:302] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:28.235929 16560 raft_consensus.cc:2781] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:28.236145 16560 raft_consensus.cc:686] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Becoming Leader. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Running, Role: LEADER
I20240502 14:00:28.236438 16560 consensus_queue.cc:237] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:28.239391 15429 catalog_manager.cc:5484] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132). New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } }
I20240502 14:00:28.253259 16569 raft_consensus.cc:482] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:28.253387 16569 raft_consensus.cc:504] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:28.254390 16569 leader_election.cc:288] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:28.257768 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81f14b7c1d1468196d0fe6d67fe93a7" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf" is_pre_election: true
I20240502 14:00:28.257792 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81f14b7c1d1468196d0fe6d67fe93a7" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917" is_pre_election: true
I20240502 14:00:28.257885 15833 raft_consensus.cc:2454] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 0.
I20240502 14:00:28.257920 15702 raft_consensus.cc:2454] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 0.
I20240502 14:00:28.258643 15507 leader_election.cc:302] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:28.258903 16569 raft_consensus.cc:2781] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:28.258962 16569 raft_consensus.cc:482] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:28.258989 16569 raft_consensus.cc:3037] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.259811 16569 raft_consensus.cc:504] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:28.260141 16569 leader_election.cc:288] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 election: Requested vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:28.260536 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81f14b7c1d1468196d0fe6d67fe93a7" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917"
I20240502 14:00:28.260536 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d81f14b7c1d1468196d0fe6d67fe93a7" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf"
I20240502 14:00:28.260635 15702 raft_consensus.cc:3037] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.260645 15833 raft_consensus.cc:3037] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.261355 15702 raft_consensus.cc:2454] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 1.
I20240502 14:00:28.261384 15833 raft_consensus.cc:2454] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 1.
I20240502 14:00:28.261627 15504 leader_election.cc:302] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:28.261811 16569 raft_consensus.cc:2781] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:28.261994 16569 raft_consensus.cc:686] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Becoming Leader. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Running, Role: LEADER
I20240502 14:00:28.262344 16569 consensus_queue.cc:237] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:28.265340 15429 catalog_manager.cc:5484] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129). New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:28.376335 16561 consensus_queue.cc:1035] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.380496 16586 consensus_queue.cc:1035] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:00:28.448457 16583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:28.448704 16583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:28.451751 16583 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 14:00:28.539942 15833 raft_consensus.cc:1261] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:28.540674 16581 consensus_queue.cc:1035] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.542127 15702 raft_consensus.cc:1261] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:28.545295 16581 consensus_queue.cc:1035] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.554261 15701 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:28.554787 16560 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.555583 15832 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:28.556605 16560 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.563587 16589 mvcc.cc:205] Tried to move back new op lower bound from 7023240923288768512 to 7023240922165055488. Current Snapshot: MvccSnapshot[applied={T|T < 7023240923288768512}]
I20240502 14:00:28.565645 16585 mvcc.cc:205] Tried to move back new op lower bound from 7023240923288768512 to 7023240922165055488. Current Snapshot: MvccSnapshot[applied={T|T < 7023240923288768512}]
I20240502 14:00:28.571931 16587 mvcc.cc:205] Tried to move back new op lower bound from 7023240923353243648 to 7023240922058743808. Current Snapshot: MvccSnapshot[applied={T|T < 7023240923353243648}]
I20240502 14:00:28.581151 16625 mvcc.cc:205] Tried to move back new op lower bound from 7023240923288768512 to 7023240922165055488. Current Snapshot: MvccSnapshot[applied={T|T < 7023240923288768512}]
INSERT report
rows total: 2000
time total: 113.173 ms
time per row: 0.0565863 ms
I20240502 14:00:28.613646 15421 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49520:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:28.619338 15550 tablet_service.cc:1467] Processing CreateTablet for tablet 387bdd8ae3b34f588e2170b061a5bc05 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a00d0c0f593149dcb05a21e3d90dda57]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:28.619587 15943 tablet_service.cc:1467] Processing CreateTablet for tablet 619a8ec66ba74ecea689c17b4734253a (DEFAULT_TABLE table=rebalance_test_table_1 [id=a00d0c0f593149dcb05a21e3d90dda57]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:28.619714 15550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387bdd8ae3b34f588e2170b061a5bc05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.619866 15943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619a8ec66ba74ecea689c17b4734253a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.620358 15811 tablet_service.cc:1467] Processing CreateTablet for tablet 02d3da05d82340178e27218c2ce56b67 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a00d0c0f593149dcb05a21e3d90dda57]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:28.620658 15811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d3da05d82340178e27218c2ce56b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.620692 15812 tablet_service.cc:1467] Processing CreateTablet for tablet 619a8ec66ba74ecea689c17b4734253a (DEFAULT_TABLE table=rebalance_test_table_1 [id=a00d0c0f593149dcb05a21e3d90dda57]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:28.620889 15812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619a8ec66ba74ecea689c17b4734253a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.619590 15944 tablet_service.cc:1467] Processing CreateTablet for tablet 387bdd8ae3b34f588e2170b061a5bc05 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a00d0c0f593149dcb05a21e3d90dda57]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:28.620352 15813 tablet_service.cc:1467] Processing CreateTablet for tablet 387bdd8ae3b34f588e2170b061a5bc05 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a00d0c0f593149dcb05a21e3d90dda57]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:28.622218 15944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387bdd8ae3b34f588e2170b061a5bc05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.622349 15813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387bdd8ae3b34f588e2170b061a5bc05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.622660 15681 tablet_service.cc:1467] Processing CreateTablet for tablet 02d3da05d82340178e27218c2ce56b67 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a00d0c0f593149dcb05a21e3d90dda57]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:28.622926 15681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d3da05d82340178e27218c2ce56b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.623641 15551 tablet_service.cc:1467] Processing CreateTablet for tablet 02d3da05d82340178e27218c2ce56b67 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a00d0c0f593149dcb05a21e3d90dda57]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:28.623878 15551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d3da05d82340178e27218c2ce56b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.624825 16651 tablet_bootstrap.cc:492] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:28.625142 16649 tablet_bootstrap.cc:492] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:28.625622 16652 tablet_bootstrap.cc:492] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:28.625856 16651 tablet_bootstrap.cc:654] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.626066 16649 tablet_bootstrap.cc:654] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.626542 16652 tablet_bootstrap.cc:654] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.626798 16650 tablet_bootstrap.cc:492] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:28.627168 15550 tablet_service.cc:1467] Processing CreateTablet for tablet 619a8ec66ba74ecea689c17b4734253a (DEFAULT_TABLE table=rebalance_test_table_1 [id=a00d0c0f593149dcb05a21e3d90dda57]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:28.627385 15550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619a8ec66ba74ecea689c17b4734253a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:28.627828 16652 tablet_bootstrap.cc:492] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:28.627919 16652 ts_tablet_manager.cc:1367] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:28.627941 16650 tablet_bootstrap.cc:654] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.628458 16652 raft_consensus.cc:348] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:28.628587 16652 raft_consensus.cc:374] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.628631 16652 raft_consensus.cc:729] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.628767 16652 consensus_queue.cc:260] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:28.628993 16649 tablet_bootstrap.cc:492] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:28.629047 16652 ts_tablet_manager.cc:1398] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:28.629081 16649 ts_tablet_manager.cc:1367] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:00:28.629917 16649 raft_consensus.cc:348] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:28.630168 16651 tablet_bootstrap.cc:492] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:28.630147 16649 raft_consensus.cc:374] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.630215 16649 raft_consensus.cc:729] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.630265 16651 ts_tablet_manager.cc:1367] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:28.630374 16649 consensus_queue.cc:260] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:28.630779 16651 raft_consensus.cc:348] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:28.630895 16651 raft_consensus.cc:374] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.630931 16651 raft_consensus.cc:729] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.631095 16651 consensus_queue.cc:260] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:28.631379 16651 ts_tablet_manager.cc:1398] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:28.631534 16651 tablet_bootstrap.cc:492] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:28.632108 16650 tablet_bootstrap.cc:492] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:28.632200 16650 ts_tablet_manager.cc:1367] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:28.632469 16651 tablet_bootstrap.cc:654] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.632728 16650 raft_consensus.cc:348] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:28.632849 16650 raft_consensus.cc:374] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.632881 16650 raft_consensus.cc:729] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.633069 16649 ts_tablet_manager.cc:1398] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:00:28.633257 16649 tablet_bootstrap.cc:492] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:28.633955 16649 tablet_bootstrap.cc:654] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.634406 16650 consensus_queue.cc:260] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:28.634805 16650 ts_tablet_manager.cc:1398] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:28.635003 16650 tablet_bootstrap.cc:492] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:28.635107 16649 tablet_bootstrap.cc:492] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:28.635175 16649 ts_tablet_manager.cc:1367] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:00:28.635620 16649 raft_consensus.cc:348] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:28.635735 16649 raft_consensus.cc:374] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.635778 16649 raft_consensus.cc:729] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.635869 16650 tablet_bootstrap.cc:654] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.635927 16649 consensus_queue.cc:260] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:28.636240 16649 ts_tablet_manager.cc:1398] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:28.636427 16649 tablet_bootstrap.cc:492] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:28.637043 16650 tablet_bootstrap.cc:492] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:28.637135 16650 ts_tablet_manager.cc:1367] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:28.637321 16649 tablet_bootstrap.cc:654] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:28.637598 16650 raft_consensus.cc:348] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:28.637722 16650 raft_consensus.cc:374] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.637768 16650 raft_consensus.cc:729] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.637914 16650 consensus_queue.cc:260] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:28.638260 16650 ts_tablet_manager.cc:1398] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:28.638679 16649 tablet_bootstrap.cc:492] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:28.638774 16649 ts_tablet_manager.cc:1367] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:00:28.639187 16649 raft_consensus.cc:348] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:28.639317 16649 raft_consensus.cc:374] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.639380 16649 raft_consensus.cc:729] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.639511 16649 consensus_queue.cc:260] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:28.639833 16649 ts_tablet_manager.cc:1398] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:28.640333 16651 tablet_bootstrap.cc:492] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:28.640444 16635 raft_consensus.cc:482] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:28.640579 16635 raft_consensus.cc:504] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:28.642148 16635 leader_election.cc:288] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:28.642699 16651 ts_tablet_manager.cc:1367] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 14:00:28.642897 16569 raft_consensus.cc:482] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:28.642971 16569 raft_consensus.cc:504] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:28.643188 16651 raft_consensus.cc:348] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:28.643277 16651 raft_consensus.cc:374] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.643312 16651 raft_consensus.cc:729] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.643379 16569 leader_election.cc:288] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719), 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:28.643468 16651 consensus_queue.cc:260] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:28.643965 16651 ts_tablet_manager.cc:1398] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:28.644151 16651 tablet_bootstrap.cc:492] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:28.644291 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619a8ec66ba74ecea689c17b4734253a" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf" is_pre_election: true
I20240502 14:00:28.645025 16651 tablet_bootstrap.cc:654] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
W20240502 14:00:28.645550 15507 leader_election.cc:341] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:00:28.646991 16651 tablet_bootstrap.cc:492] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:28.647096 16651 ts_tablet_manager.cc:1367] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:28.647493 16651 raft_consensus.cc:348] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:28.647632 16651 raft_consensus.cc:374] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:28.647689 16651 raft_consensus.cc:729] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.647826 16651 consensus_queue.cc:260] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:28.648123 16651 ts_tablet_manager.cc:1398] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:28.649387 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619a8ec66ba74ecea689c17b4734253a" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" is_pre_election: true
I20240502 14:00:28.649538 15964 raft_consensus.cc:2454] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 0.
I20240502 14:00:28.649837 15504 leader_election.cc:302] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 82d7b7201cfc431d90baca1ad0a397cf
I20240502 14:00:28.650023 16569 raft_consensus.cc:2781] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:28.650080 16569 raft_consensus.cc:482] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:28.650132 16569 raft_consensus.cc:3037] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.649843 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d3da05d82340178e27218c2ce56b67" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" is_pre_election: true
I20240502 14:00:28.650228 15571 raft_consensus.cc:2454] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 0.
I20240502 14:00:28.651124 15768 leader_election.cc:302] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:28.651158 16569 raft_consensus.cc:504] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:28.651186 15701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d3da05d82340178e27218c2ce56b67" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917" is_pre_election: true
I20240502 14:00:28.651329 15701 raft_consensus.cc:2454] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 0.
I20240502 14:00:28.651371 16635 raft_consensus.cc:2781] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:28.651420 16635 raft_consensus.cc:482] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:28.651475 16635 raft_consensus.cc:3037] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.651546 16569 leader_election.cc:288] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 election: Requested vote from peers 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719), 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:28.652283 16635 raft_consensus.cc:504] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:28.652552 16635 leader_election.cc:288] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 election: Requested vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:28.652915 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d3da05d82340178e27218c2ce56b67" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
I20240502 14:00:28.652958 15701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d3da05d82340178e27218c2ce56b67" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917"
I20240502 14:00:28.653015 15571 raft_consensus.cc:3037] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.653079 15701 raft_consensus.cc:3037] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.653132 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619a8ec66ba74ecea689c17b4734253a" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf"
I20240502 14:00:28.653230 15833 raft_consensus.cc:3037] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.653851 15571 raft_consensus.cc:2454] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 1.
I20240502 14:00:28.654009 15701 raft_consensus.cc:2454] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 1.
I20240502 14:00:28.654009 15833 raft_consensus.cc:2454] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 1.
I20240502 14:00:28.654124 15768 leader_election.cc:302] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:28.654352 16635 raft_consensus.cc:2781] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:28.654443 16635 raft_consensus.cc:686] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Becoming Leader. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Running, Role: LEADER
I20240502 14:00:28.654423 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619a8ec66ba74ecea689c17b4734253a" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
I20240502 14:00:28.654567 15964 raft_consensus.cc:3037] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.654567 15507 leader_election.cc:302] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:28.654731 16635 consensus_queue.cc:237] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:28.654841 16569 raft_consensus.cc:2781] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:28.654930 16569 raft_consensus.cc:686] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Becoming Leader. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Running, Role: LEADER
I20240502 14:00:28.655053 16569 consensus_queue.cc:237] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:28.655498 15964 raft_consensus.cc:2454] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 1.
I20240502 14:00:28.656715 15421 catalog_manager.cc:5484] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129). New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:28.657178 15429 catalog_manager.cc:5484] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131). New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:28.780076 16577 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:28.780197 16577 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:28.780660 16577 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237), 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:28.781077 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387bdd8ae3b34f588e2170b061a5bc05" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf" is_pre_election: true
I20240502 14:00:28.781239 15833 raft_consensus.cc:2454] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 0.
I20240502 14:00:28.781522 15900 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:28.781662 16577 raft_consensus.cc:2781] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:28.781724 16577 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:28.781796 16577 raft_consensus.cc:3037] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.782925 16577 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:28.783296 16577 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 election: Requested vote from peers 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237), 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:28.783618 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387bdd8ae3b34f588e2170b061a5bc05" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf"
I20240502 14:00:28.783784 15833 raft_consensus.cc:3037] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:28.784528 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387bdd8ae3b34f588e2170b061a5bc05" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" is_pre_election: true
I20240502 14:00:28.784591 15833 raft_consensus.cc:2454] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 1.
I20240502 14:00:28.784632 15570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387bdd8ae3b34f588e2170b061a5bc05" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
I20240502 14:00:28.784684 15571 raft_consensus.cc:2454] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 0.
I20240502 14:00:28.784878 15900 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:28.785045 16577 raft_consensus.cc:2781] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:28.785146 16577 raft_consensus.cc:686] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Becoming Leader. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Running, Role: LEADER
I20240502 14:00:28.785291 16577 consensus_queue.cc:237] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:28.786820 15429 catalog_manager.cc:5484] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132). New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } }
W20240502 14:00:28.917948 16668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:28.918156 16668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:28.921217 16668 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 14:00:28.998005 15571 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:28.998425 15964 raft_consensus.cc:1261] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:28.998977 16569 consensus_queue.cc:1035] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:28.999373 15833 raft_consensus.cc:1261] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:28.999912 15833 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:28.999995 15701 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:29.000386 16635 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:29.000468 16577 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:29.001080 15571 raft_consensus.cc:3037] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.002154 15571 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:29.002743 16581 consensus_queue.cc:1035] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:29.003840 16560 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:29.004338 16635 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 108.676 ms
time per row: 0.0543381 ms
I20240502 14:00:29.070756 15427 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49520:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:29.075888 15681 tablet_service.cc:1467] Processing CreateTablet for tablet 1f562ad00ec34a8d9184671ed55d84d1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b36f1f16cb3d4b8bae9303e2976dc2e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.075903 15944 tablet_service.cc:1467] Processing CreateTablet for tablet 1f562ad00ec34a8d9184671ed55d84d1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b36f1f16cb3d4b8bae9303e2976dc2e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.075973 15550 tablet_service.cc:1467] Processing CreateTablet for tablet 1f562ad00ec34a8d9184671ed55d84d1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b36f1f16cb3d4b8bae9303e2976dc2e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.076203 15550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f562ad00ec34a8d9184671ed55d84d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.076208 15944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f562ad00ec34a8d9184671ed55d84d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.076206 15681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f562ad00ec34a8d9184671ed55d84d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.077481 15812 tablet_service.cc:1467] Processing CreateTablet for tablet 48fbc670ec504d26a11bf1b9fbd78981 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b36f1f16cb3d4b8bae9303e2976dc2e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.077746 15812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48fbc670ec504d26a11bf1b9fbd78981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.078254 16650 tablet_bootstrap.cc:492] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:29.078410 16649 tablet_bootstrap.cc:492] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:29.078471 16652 tablet_bootstrap.cc:492] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:29.079182 16650 tablet_bootstrap.cc:654] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.079191 16649 tablet_bootstrap.cc:654] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.079352 16652 tablet_bootstrap.cc:654] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.079871 16651 tablet_bootstrap.cc:492] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:29.080189 16650 tablet_bootstrap.cc:492] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:29.080264 16650 ts_tablet_manager.cc:1367] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:29.080272 15550 tablet_service.cc:1467] Processing CreateTablet for tablet a7981c3f46d8483caae82fbf2296e762 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b36f1f16cb3d4b8bae9303e2976dc2e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.080478 15550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7981c3f46d8483caae82fbf2296e762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.080750 16651 tablet_bootstrap.cc:654] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.080731 16650 raft_consensus.cc:348] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.080830 16650 raft_consensus.cc:374] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.080859 16650 raft_consensus.cc:729] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.080967 16650 consensus_queue.cc:260] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.081231 16650 ts_tablet_manager.cc:1398] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.081804 16651 tablet_bootstrap.cc:492] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:29.081887 16651 ts_tablet_manager.cc:1367] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:29.081985 16652 tablet_bootstrap.cc:492] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:29.082063 16652 ts_tablet_manager.cc:1367] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:29.082368 16651 raft_consensus.cc:348] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:29.082469 16651 raft_consensus.cc:374] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.082501 16651 raft_consensus.cc:729] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.082526 16652 raft_consensus.cc:348] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.082625 16652 raft_consensus.cc:374] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.082657 16652 raft_consensus.cc:729] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.082634 16651 consensus_queue.cc:260] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:29.082783 16652 consensus_queue.cc:260] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.082935 16651 ts_tablet_manager.cc:1398] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.082950 15943 tablet_service.cc:1467] Processing CreateTablet for tablet a7981c3f46d8483caae82fbf2296e762 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b36f1f16cb3d4b8bae9303e2976dc2e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.083034 16652 ts_tablet_manager.cc:1398] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.083180 15943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7981c3f46d8483caae82fbf2296e762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.083314 15682 tablet_service.cc:1467] Processing CreateTablet for tablet a7981c3f46d8483caae82fbf2296e762 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b36f1f16cb3d4b8bae9303e2976dc2e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.083546 15682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7981c3f46d8483caae82fbf2296e762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.084314 16649 tablet_bootstrap.cc:492] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:29.084399 16649 ts_tablet_manager.cc:1367] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240502 14:00:29.084829 16649 raft_consensus.cc:348] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.084940 16649 raft_consensus.cc:374] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.084972 16649 raft_consensus.cc:729] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.085094 16649 consensus_queue.cc:260] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.085242 16650 tablet_bootstrap.cc:492] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:29.085386 16649 ts_tablet_manager.cc:1398] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:29.085564 16649 tablet_bootstrap.cc:492] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:29.085566 15681 tablet_service.cc:1467] Processing CreateTablet for tablet 48fbc670ec504d26a11bf1b9fbd78981 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b36f1f16cb3d4b8bae9303e2976dc2e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.085775 15681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48fbc670ec504d26a11bf1b9fbd78981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.086176 16650 tablet_bootstrap.cc:654] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.086483 16652 tablet_bootstrap.cc:492] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:29.087364 16652 tablet_bootstrap.cc:654] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.088101 15944 tablet_service.cc:1467] Processing CreateTablet for tablet 48fbc670ec504d26a11bf1b9fbd78981 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b36f1f16cb3d4b8bae9303e2976dc2e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.088330 15944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48fbc670ec504d26a11bf1b9fbd78981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.089073 16652 tablet_bootstrap.cc:492] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:29.089174 16652 ts_tablet_manager.cc:1367] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:29.089248 16650 tablet_bootstrap.cc:492] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:29.089347 16650 ts_tablet_manager.cc:1367] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:29.089632 16652 raft_consensus.cc:348] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.089736 16652 raft_consensus.cc:374] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.089769 16652 raft_consensus.cc:729] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.089727 16650 raft_consensus.cc:348] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.089846 16650 raft_consensus.cc:374] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.089891 16650 raft_consensus.cc:729] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.090039 16650 consensus_queue.cc:260] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.090225 16652 consensus_queue.cc:260] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.090555 16649 tablet_bootstrap.cc:654] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.090415 16650 ts_tablet_manager.cc:1398] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.090699 16652 ts_tablet_manager.cc:1398] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.090719 16650 tablet_bootstrap.cc:492] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:29.091038 16652 tablet_bootstrap.cc:492] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:29.091902 16650 tablet_bootstrap.cc:654] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.092458 16649 tablet_bootstrap.cc:492] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:29.092533 16649 ts_tablet_manager.cc:1367] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:29.092703 16652 tablet_bootstrap.cc:654] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.093050 16649 raft_consensus.cc:348] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.093169 16649 raft_consensus.cc:374] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.093212 16649 raft_consensus.cc:729] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.093258 16650 tablet_bootstrap.cc:492] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:29.093338 16650 ts_tablet_manager.cc:1367] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:29.093328 16649 consensus_queue.cc:260] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.093613 16649 ts_tablet_manager.cc:1398] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.093734 16650 raft_consensus.cc:348] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:29.093832 16650 raft_consensus.cc:374] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.093864 16650 raft_consensus.cc:729] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.093987 16650 consensus_queue.cc:260] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:29.094269 16650 ts_tablet_manager.cc:1398] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:29.094425 16652 tablet_bootstrap.cc:492] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:29.094496 16652 ts_tablet_manager.cc:1367] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:29.094914 16652 raft_consensus.cc:348] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:29.095009 16652 raft_consensus.cc:374] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.095045 16652 raft_consensus.cc:729] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.095142 16652 consensus_queue.cc:260] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:29.095465 16652 ts_tablet_manager.cc:1398] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:29.150580 16561 raft_consensus.cc:482] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:29.150689 16561 raft_consensus.cc:504] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:29.151135 16561 leader_election.cc:288] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719), 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:29.151479 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48fbc670ec504d26a11bf1b9fbd78981" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" is_pre_election: true
I20240502 14:00:29.151624 15964 raft_consensus.cc:2454] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 0.
I20240502 14:00:29.151954 15635 leader_election.cc:302] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:29.152127 16561 raft_consensus.cc:2781] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:29.152186 16561 raft_consensus.cc:482] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:29.152230 16561 raft_consensus.cc:3037] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.153151 16561 raft_consensus.cc:504] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:29.153450 16561 leader_election.cc:288] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 election: Requested vote from peers 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719), 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:29.153595 16569 raft_consensus.cc:482] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:29.153740 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48fbc670ec504d26a11bf1b9fbd78981" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
I20240502 14:00:29.153715 16569 raft_consensus.cc:504] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.153849 15964 raft_consensus.cc:3037] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.154134 16569 leader_election.cc:288] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:29.154589 15963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f562ad00ec34a8d9184671ed55d84d1" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" is_pre_election: true
I20240502 14:00:29.154588 15701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f562ad00ec34a8d9184671ed55d84d1" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917" is_pre_election: true
I20240502 14:00:29.154695 15963 raft_consensus.cc:2454] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 0.
I20240502 14:00:29.154695 15701 raft_consensus.cc:2454] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 0.
I20240502 14:00:29.154752 15964 raft_consensus.cc:2454] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 1.
I20240502 14:00:29.154966 15504 leader_election.cc:302] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 4d0f8f311e834d81bd54b9f8e7923bcd; no voters:
I20240502 14:00:29.155028 15635 leader_election.cc:302] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:29.155159 16561 raft_consensus.cc:2781] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:29.155175 16569 raft_consensus.cc:2781] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:29.155225 16569 raft_consensus.cc:482] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:29.155259 16569 raft_consensus.cc:3037] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.155309 16561 raft_consensus.cc:686] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Becoming Leader. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Running, Role: LEADER
I20240502 14:00:29.155436 16561 consensus_queue.cc:237] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:29.156034 15832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48fbc670ec504d26a11bf1b9fbd78981" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf" is_pre_election: true
I20240502 14:00:29.156108 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48fbc670ec504d26a11bf1b9fbd78981" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf"
I20240502 14:00:29.156109 16569 raft_consensus.cc:504] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.156160 15832 raft_consensus.cc:2454] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 0.
I20240502 14:00:29.156445 16569 leader_election.cc:288] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 election: Requested vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:29.156735 15701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f562ad00ec34a8d9184671ed55d84d1" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917"
I20240502 14:00:29.156838 15701 raft_consensus.cc:3037] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.156814 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f562ad00ec34a8d9184671ed55d84d1" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
I20240502 14:00:29.156904 15964 raft_consensus.cc:3037] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.157655 15701 raft_consensus.cc:2454] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 1.
I20240502 14:00:29.157660 15964 raft_consensus.cc:2454] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 1.
I20240502 14:00:29.157905 15504 leader_election.cc:302] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:29.158090 16569 raft_consensus.cc:2781] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:29.158201 16569 raft_consensus.cc:686] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Becoming Leader. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Running, Role: LEADER
I20240502 14:00:29.158360 16569 consensus_queue.cc:237] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.158310 15427 catalog_manager.cc:5484] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8484181c4ece483c8b408817fab1a917 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:29.160746 15428 catalog_manager.cc:5484] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129). New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } }
I20240502 14:00:29.235386 16561 raft_consensus.cc:482] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:29.235558 16561 raft_consensus.cc:504] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.235903 16561 leader_election.cc:288] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719), 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:29.236296 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7981c3f46d8483caae82fbf2296e762" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" is_pre_election: true
I20240502 14:00:29.236321 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7981c3f46d8483caae82fbf2296e762" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" is_pre_election: true
I20240502 14:00:29.236439 15964 raft_consensus.cc:2454] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 0.
I20240502 14:00:29.236472 15571 raft_consensus.cc:2454] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 0.
I20240502 14:00:29.236737 15635 leader_election.cc:302] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:29.236944 16561 raft_consensus.cc:2781] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:29.237015 16561 raft_consensus.cc:482] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:29.237042 16561 raft_consensus.cc:3037] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.237820 16561 raft_consensus.cc:504] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.238142 16561 leader_election.cc:288] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 election: Requested vote from peers 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719), 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:29.238507 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7981c3f46d8483caae82fbf2296e762" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
I20240502 14:00:29.238526 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7981c3f46d8483caae82fbf2296e762" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
I20240502 14:00:29.238627 15571 raft_consensus.cc:3037] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.238633 15964 raft_consensus.cc:3037] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.239320 15571 raft_consensus.cc:2454] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 1.
I20240502 14:00:29.239372 15964 raft_consensus.cc:2454] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 1.
I20240502 14:00:29.239603 15637 leader_election.cc:302] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:29.239785 16561 raft_consensus.cc:2781] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:29.239894 16561 raft_consensus.cc:686] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Becoming Leader. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Running, Role: LEADER
I20240502 14:00:29.240018 16561 consensus_queue.cc:237] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.241640 15428 catalog_manager.cc:5484] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8484181c4ece483c8b408817fab1a917 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } }
W20240502 14:00:29.374754 16717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:29.374982 16717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:29.377925 16717 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 14:00:29.457583 15701 raft_consensus.cc:1261] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:29.457765 15964 raft_consensus.cc:1261] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:29.458374 15833 raft_consensus.cc:3037] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.458452 16581 consensus_queue.cc:1035] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:29.458693 15964 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:29.458762 16569 consensus_queue.cc:1035] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:29.459124 16713 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:29.459581 15833 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:29.459909 15571 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:29.460371 16742 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:29.460675 16742 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:29.464756 15964 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:29.465234 16713 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.811 ms
time per row: 0.0559057 ms
I20240502 14:00:29.533193 15423 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49520:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:29.538857 15550 tablet_service.cc:1467] Processing CreateTablet for tablet 1f80c8590e994a10a9b7c6c174e28173 (DEFAULT_TABLE table=rebalance_test_table_3 [id=12389884a03b40f68c532a06ec761e9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.539040 15813 tablet_service.cc:1467] Processing CreateTablet for tablet b6d365c10c76416a944ec8fe409e3dc5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=12389884a03b40f68c532a06ec761e9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.539201 15550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f80c8590e994a10a9b7c6c174e28173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.539332 15813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d365c10c76416a944ec8fe409e3dc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.539359 15812 tablet_service.cc:1467] Processing CreateTablet for tablet 1f80c8590e994a10a9b7c6c174e28173 (DEFAULT_TABLE table=rebalance_test_table_3 [id=12389884a03b40f68c532a06ec761e9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.539544 15812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f80c8590e994a10a9b7c6c174e28173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.539672 15551 tablet_service.cc:1467] Processing CreateTablet for tablet 28e371c1a999431499e09479e8f5b44b (DEFAULT_TABLE table=rebalance_test_table_3 [id=12389884a03b40f68c532a06ec761e9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.539865 15551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e371c1a999431499e09479e8f5b44b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.540760 15681 tablet_service.cc:1467] Processing CreateTablet for tablet 1f80c8590e994a10a9b7c6c174e28173 (DEFAULT_TABLE table=rebalance_test_table_3 [id=12389884a03b40f68c532a06ec761e9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.541013 15681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f80c8590e994a10a9b7c6c174e28173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.541465 16651 tablet_bootstrap.cc:492] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:29.541539 16649 tablet_bootstrap.cc:492] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:29.542398 16651 tablet_bootstrap.cc:654] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.542519 16649 tablet_bootstrap.cc:654] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.543447 16652 tablet_bootstrap.cc:492] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:29.543555 15813 tablet_service.cc:1467] Processing CreateTablet for tablet 28e371c1a999431499e09479e8f5b44b (DEFAULT_TABLE table=rebalance_test_table_3 [id=12389884a03b40f68c532a06ec761e9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.543766 15813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e371c1a999431499e09479e8f5b44b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.544379 16652 tablet_bootstrap.cc:654] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.545226 15944 tablet_service.cc:1467] Processing CreateTablet for tablet b6d365c10c76416a944ec8fe409e3dc5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=12389884a03b40f68c532a06ec761e9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.545537 15944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d365c10c76416a944ec8fe409e3dc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.545943 16649 tablet_bootstrap.cc:492] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:29.546032 16649 ts_tablet_manager.cc:1367] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:29.546500 16649 raft_consensus.cc:348] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.546650 16649 raft_consensus.cc:374] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.546694 16649 raft_consensus.cc:729] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.546820 16649 consensus_queue.cc:260] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.547091 16649 ts_tablet_manager.cc:1398] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.547399 16649 tablet_bootstrap.cc:492] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:29.547425 16651 tablet_bootstrap.cc:492] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:29.547508 16651 ts_tablet_manager.cc:1367] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240502 14:00:29.547924 15943 tablet_service.cc:1467] Processing CreateTablet for tablet 28e371c1a999431499e09479e8f5b44b (DEFAULT_TABLE table=rebalance_test_table_3 [id=12389884a03b40f68c532a06ec761e9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.547981 16651 raft_consensus.cc:348] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:29.548074 16651 raft_consensus.cc:374] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.548105 16651 raft_consensus.cc:729] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.548169 15943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e371c1a999431499e09479e8f5b44b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.548278 16649 tablet_bootstrap.cc:654] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.548240 16651 consensus_queue.cc:260] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:29.548523 16651 ts_tablet_manager.cc:1398] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:29.548676 16651 tablet_bootstrap.cc:492] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:29.549736 16651 tablet_bootstrap.cc:654] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.549996 16650 tablet_bootstrap.cc:492] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:29.551221 16650 tablet_bootstrap.cc:654] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.551887 16651 tablet_bootstrap.cc:492] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:29.551940 16651 ts_tablet_manager.cc:1367] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:00:29.552347 16651 raft_consensus.cc:348] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.552398 16652 tablet_bootstrap.cc:492] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:29.552433 16651 raft_consensus.cc:374] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.552465 16651 raft_consensus.cc:729] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.552488 16652 ts_tablet_manager.cc:1367] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240502 14:00:29.552593 16651 consensus_queue.cc:260] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.552891 16651 ts_tablet_manager.cc:1398] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.553036 16651 tablet_bootstrap.cc:492] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:29.552994 16652 raft_consensus.cc:348] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.553113 16652 raft_consensus.cc:374] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.553160 16652 raft_consensus.cc:729] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.553295 16652 consensus_queue.cc:260] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.553603 16652 ts_tablet_manager.cc:1398] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 14:00:29.553741 16650 tablet_bootstrap.cc:492] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:29.553817 16650 ts_tablet_manager.cc:1367] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:00:29.553885 16651 tablet_bootstrap.cc:654] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.554112 15681 tablet_service.cc:1467] Processing CreateTablet for tablet b6d365c10c76416a944ec8fe409e3dc5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=12389884a03b40f68c532a06ec761e9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:29.554245 16650 raft_consensus.cc:348] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:29.554373 16650 raft_consensus.cc:374] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.554399 15681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d365c10c76416a944ec8fe409e3dc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.554406 16650 raft_consensus.cc:729] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.554543 16650 consensus_queue.cc:260] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:29.554805 16650 ts_tablet_manager.cc:1398] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.554948 16650 tablet_bootstrap.cc:492] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:29.556514 16652 tablet_bootstrap.cc:492] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:29.557571 16650 tablet_bootstrap.cc:654] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.557827 16651 tablet_bootstrap.cc:492] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:29.557857 16652 tablet_bootstrap.cc:654] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.557914 16651 ts_tablet_manager.cc:1367] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:00:29.558363 16651 raft_consensus.cc:348] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.558477 16651 raft_consensus.cc:374] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.558501 16649 tablet_bootstrap.cc:492] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:29.558517 16651 raft_consensus.cc:729] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.558586 16649 ts_tablet_manager.cc:1367] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 14:00:29.558642 16651 consensus_queue.cc:260] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.559010 16651 ts_tablet_manager.cc:1398] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:29.559291 16652 tablet_bootstrap.cc:492] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:29.559264 16649 raft_consensus.cc:348] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.559376 16652 ts_tablet_manager.cc:1367] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:29.559384 16649 raft_consensus.cc:374] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.559425 16649 raft_consensus.cc:729] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.559568 16649 consensus_queue.cc:260] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.559831 16649 ts_tablet_manager.cc:1398] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.559793 16652 raft_consensus.cc:348] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:29.559885 16652 raft_consensus.cc:374] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.559916 16652 raft_consensus.cc:729] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.560025 16652 consensus_queue.cc:260] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:29.560281 16652 ts_tablet_manager.cc:1398] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.561846 16650 tablet_bootstrap.cc:492] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:29.561947 16650 ts_tablet_manager.cc:1367] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 14:00:29.562250 16635 raft_consensus.cc:482] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:29.562350 16635 raft_consensus.cc:504] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.562469 16650 raft_consensus.cc:348] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.562593 16650 raft_consensus.cc:374] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.562637 16650 raft_consensus.cc:729] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.562768 16650 consensus_queue.cc:260] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.562845 16635 leader_election.cc:288] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719), 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:29.563068 16650 ts_tablet_manager.cc:1398] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:29.563563 16755 raft_consensus.cc:482] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:29.563657 16755 raft_consensus.cc:504] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.564044 16755 leader_election.cc:288] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531), 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:29.564071 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e371c1a999431499e09479e8f5b44b" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" is_pre_election: true
I20240502 14:00:29.564183 15571 raft_consensus.cc:2454] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 0.
I20240502 14:00:29.564349 15570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f80c8590e994a10a9b7c6c174e28173" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" is_pre_election: true
I20240502 14:00:29.564443 15570 raft_consensus.cc:2454] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 0.
I20240502 14:00:29.564496 15768 leader_election.cc:302] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:29.564661 16635 raft_consensus.cc:2781] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:29.564709 16635 raft_consensus.cc:482] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:29.564728 15637 leader_election.cc:302] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:29.564755 16635 raft_consensus.cc:3037] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.564877 16755 raft_consensus.cc:2781] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:29.564929 16755 raft_consensus.cc:482] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:29.564965 16755 raft_consensus.cc:3037] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.564929 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f80c8590e994a10a9b7c6c174e28173" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf" is_pre_election: true
I20240502 14:00:29.565078 15833 raft_consensus.cc:2454] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 0.
I20240502 14:00:29.565697 16635 raft_consensus.cc:504] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.565888 16755 raft_consensus.cc:504] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.566072 16635 leader_election.cc:288] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 election: Requested vote from peers 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719), 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:29.566216 16755 leader_election.cc:288] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 election: Requested vote from peers 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531), 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:29.566622 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f80c8590e994a10a9b7c6c174e28173" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
I20240502 14:00:29.566732 15570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e371c1a999431499e09479e8f5b44b" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
I20240502 14:00:29.566746 15571 raft_consensus.cc:3037] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.566823 15570 raft_consensus.cc:3037] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.567606 15571 raft_consensus.cc:2454] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 1.
I20240502 14:00:29.567670 15570 raft_consensus.cc:2454] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 1.
I20240502 14:00:29.567909 15637 leader_election.cc:302] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:29.568060 15768 leader_election.cc:302] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:29.568233 16635 raft_consensus.cc:2781] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:29.568337 16755 raft_consensus.cc:2781] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:29.568377 16635 raft_consensus.cc:686] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Becoming Leader. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Running, Role: LEADER
I20240502 14:00:29.568416 16755 raft_consensus.cc:686] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Becoming Leader. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Running, Role: LEADER
I20240502 14:00:29.568500 16635 consensus_queue.cc:237] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.568548 16755 consensus_queue.cc:237] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:29.568835 15963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e371c1a999431499e09479e8f5b44b" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" is_pre_election: true
I20240502 14:00:29.568869 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e371c1a999431499e09479e8f5b44b" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
I20240502 14:00:29.568940 15963 raft_consensus.cc:2454] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 0.
I20240502 14:00:29.568499 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f80c8590e994a10a9b7c6c174e28173" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf"
I20240502 14:00:29.569204 15833 raft_consensus.cc:3037] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.570147 15833 raft_consensus.cc:2454] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 1.
I20240502 14:00:29.570999 15428 catalog_manager.cc:5484] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131). New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:29.571350 15421 catalog_manager.cc:5484] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8484181c4ece483c8b408817fab1a917 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:29.933219 16755 raft_consensus.cc:482] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:29.933339 16755 raft_consensus.cc:504] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:29.933730 16755 leader_election.cc:288] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531), 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:29.934149 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d365c10c76416a944ec8fe409e3dc5" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" is_pre_election: true
I20240502 14:00:29.934150 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d365c10c76416a944ec8fe409e3dc5" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf" is_pre_election: true
I20240502 14:00:29.934305 15964 raft_consensus.cc:2454] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 0.
I20240502 14:00:29.934326 15833 raft_consensus.cc:2454] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 0.
I20240502 14:00:29.934616 15635 leader_election.cc:302] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:29.934809 16755 raft_consensus.cc:2781] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:29.934871 16755 raft_consensus.cc:482] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:29.934916 16755 raft_consensus.cc:3037] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.935724 16755 raft_consensus.cc:504] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:29.936033 16755 leader_election.cc:288] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 election: Requested vote from peers 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531), 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:29.936340 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d365c10c76416a944ec8fe409e3dc5" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf"
I20240502 14:00:29.936362 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d365c10c76416a944ec8fe409e3dc5" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
I20240502 14:00:29.936452 15833 raft_consensus.cc:3037] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.936466 15964 raft_consensus.cc:3037] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.937151 15833 raft_consensus.cc:2454] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 1.
I20240502 14:00:29.937230 15964 raft_consensus.cc:2454] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 1.
I20240502 14:00:29.937423 15638 leader_election.cc:302] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82d7b7201cfc431d90baca1ad0a397cf, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:29.937659 16755 raft_consensus.cc:2781] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:29.937757 16755 raft_consensus.cc:686] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Becoming Leader. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Running, Role: LEADER
I20240502 14:00:29.937896 16755 consensus_queue.cc:237] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:29.939474 15420 catalog_manager.cc:5484] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8484181c4ece483c8b408817fab1a917 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:30.022845 15964 raft_consensus.cc:3037] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.024408 16769 consensus_queue.cc:1035] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.027285 16755 consensus_queue.cc:1035] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.030845 16769 consensus_queue.cc:1035] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.033190 16756 consensus_queue.cc:1035] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:00:30.111923 16774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:30.112150 16774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:30.115058 16774 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 14:00:30.229235 15964 raft_consensus.cc:1261] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:30.229785 16756 consensus_queue.cc:1035] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.236296 15833 raft_consensus.cc:1261] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:30.236714 16756 consensus_queue.cc:1035] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 113.407 ms
time per row: 0.0567034 ms
I20240502 14:00:30.274626 15428 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49520:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:30.279664 15681 tablet_service.cc:1467] Processing CreateTablet for tablet bb9f100600a246708457280d91826fce (DEFAULT_TABLE table=rebalance_test_table_4 [id=278d2359eb234a2b8e569cd993a004a7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.279701 15551 tablet_service.cc:1467] Processing CreateTablet for tablet bb9f100600a246708457280d91826fce (DEFAULT_TABLE table=rebalance_test_table_4 [id=278d2359eb234a2b8e569cd993a004a7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.279980 15551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9f100600a246708457280d91826fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.279979 15681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9f100600a246708457280d91826fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.280031 15550 tablet_service.cc:1467] Processing CreateTablet for tablet 6e76bd0ecd3e4c168e141fb7984af3ec (DEFAULT_TABLE table=rebalance_test_table_4 [id=278d2359eb234a2b8e569cd993a004a7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.280218 15550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e76bd0ecd3e4c168e141fb7984af3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.280928 15813 tablet_service.cc:1467] Processing CreateTablet for tablet bb9f100600a246708457280d91826fce (DEFAULT_TABLE table=rebalance_test_table_4 [id=278d2359eb234a2b8e569cd993a004a7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.281188 15813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9f100600a246708457280d91826fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.282352 16830 tablet_bootstrap.cc:492] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:30.282537 15943 tablet_service.cc:1467] Processing CreateTablet for tablet 6e76bd0ecd3e4c168e141fb7984af3ec (DEFAULT_TABLE table=rebalance_test_table_4 [id=278d2359eb234a2b8e569cd993a004a7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.282792 15943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e76bd0ecd3e4c168e141fb7984af3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.283197 16830 tablet_bootstrap.cc:654] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.283408 16831 tablet_bootstrap.cc:492] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:30.283532 15812 tablet_service.cc:1467] Processing CreateTablet for tablet 2d92e5885bf94c3e8e0e4dee40017699 (DEFAULT_TABLE table=rebalance_test_table_4 [id=278d2359eb234a2b8e569cd993a004a7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.283752 15812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d92e5885bf94c3e8e0e4dee40017699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.284106 16831 tablet_bootstrap.cc:654] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.284943 16834 tablet_bootstrap.cc:492] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:30.285140 16831 tablet_bootstrap.cc:492] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:30.285113 15944 tablet_service.cc:1467] Processing CreateTablet for tablet 2d92e5885bf94c3e8e0e4dee40017699 (DEFAULT_TABLE table=rebalance_test_table_4 [id=278d2359eb234a2b8e569cd993a004a7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.285219 16831 ts_tablet_manager.cc:1367] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:30.285346 15944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d92e5885bf94c3e8e0e4dee40017699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.285655 16831 raft_consensus.cc:348] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:30.285768 16834 tablet_bootstrap.cc:654] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.285770 16831 raft_consensus.cc:374] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.285802 16831 raft_consensus.cc:729] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.285955 16831 consensus_queue.cc:260] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:30.286361 16831 ts_tablet_manager.cc:1398] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:30.286520 16831 tablet_bootstrap.cc:492] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:30.286604 16830 tablet_bootstrap.cc:492] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:30.286681 16830 ts_tablet_manager.cc:1367] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:30.286852 16834 tablet_bootstrap.cc:492] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:30.286932 16834 ts_tablet_manager.cc:1367] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:30.287073 16830 raft_consensus.cc:348] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:30.287185 16830 raft_consensus.cc:374] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.287221 16830 raft_consensus.cc:729] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.287333 16831 tablet_bootstrap.cc:654] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.287350 16830 consensus_queue.cc:260] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:30.287349 16834 raft_consensus.cc:348] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:30.287441 16834 raft_consensus.cc:374] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.287478 16834 raft_consensus.cc:729] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.287617 16830 ts_tablet_manager.cc:1398] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:30.287616 16834 consensus_queue.cc:260] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:30.287801 16830 tablet_bootstrap.cc:492] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:30.287887 16834 ts_tablet_manager.cc:1398] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:30.287938 15682 tablet_service.cc:1467] Processing CreateTablet for tablet 2d92e5885bf94c3e8e0e4dee40017699 (DEFAULT_TABLE table=rebalance_test_table_4 [id=278d2359eb234a2b8e569cd993a004a7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.288058 16834 tablet_bootstrap.cc:492] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:30.288151 15682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d92e5885bf94c3e8e0e4dee40017699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.288544 16830 tablet_bootstrap.cc:654] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.288842 16834 tablet_bootstrap.cc:654] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.289934 16830 tablet_bootstrap.cc:492] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:30.290007 16830 ts_tablet_manager.cc:1367] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:30.290542 16830 raft_consensus.cc:348] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:30.290635 16830 raft_consensus.cc:374] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.290663 16830 raft_consensus.cc:729] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.290781 16830 consensus_queue.cc:260] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:30.290899 16834 tablet_bootstrap.cc:492] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:30.290966 16834 ts_tablet_manager.cc:1367] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:30.291065 16830 ts_tablet_manager.cc:1398] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:30.291370 16834 raft_consensus.cc:348] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:30.291499 16834 raft_consensus.cc:374] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.291554 16834 raft_consensus.cc:729] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.291652 15681 tablet_service.cc:1467] Processing CreateTablet for tablet 6e76bd0ecd3e4c168e141fb7984af3ec (DEFAULT_TABLE table=rebalance_test_table_4 [id=278d2359eb234a2b8e569cd993a004a7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.291682 16834 consensus_queue.cc:260] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:30.291888 15681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e76bd0ecd3e4c168e141fb7984af3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.292232 16834 ts_tablet_manager.cc:1398] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:30.293313 16831 tablet_bootstrap.cc:492] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:30.293401 16831 ts_tablet_manager.cc:1367] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:00:30.293740 16829 tablet_bootstrap.cc:492] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:30.293843 16831 raft_consensus.cc:348] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:30.294080 16831 raft_consensus.cc:374] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.294138 16831 raft_consensus.cc:729] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.294332 16831 consensus_queue.cc:260] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:30.294638 16829 tablet_bootstrap.cc:654] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.294643 16831 ts_tablet_manager.cc:1398] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:30.295732 16829 tablet_bootstrap.cc:492] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:30.295867 16829 ts_tablet_manager.cc:1367] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:30.296368 16829 raft_consensus.cc:348] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:30.296478 16829 raft_consensus.cc:374] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.296512 16829 raft_consensus.cc:729] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.296639 16829 consensus_queue.cc:260] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:30.296931 16829 ts_tablet_manager.cc:1398] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:30.297088 16829 tablet_bootstrap.cc:492] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:30.298077 16829 tablet_bootstrap.cc:654] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.299121 16829 tablet_bootstrap.cc:492] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:30.299208 16829 ts_tablet_manager.cc:1367] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:30.299530 16829 raft_consensus.cc:348] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:30.299654 16829 raft_consensus.cc:374] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.299707 16829 raft_consensus.cc:729] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.299827 16829 consensus_queue.cc:260] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:30.300073 16829 ts_tablet_manager.cc:1398] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:30.300277 16829 tablet_bootstrap.cc:492] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:30.300979 16829 tablet_bootstrap.cc:654] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.302012 16829 tablet_bootstrap.cc:492] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:30.302076 16829 ts_tablet_manager.cc:1367] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:30.302464 16829 raft_consensus.cc:348] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:30.302548 16829 raft_consensus.cc:374] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.302570 16829 raft_consensus.cc:729] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.302693 16829 consensus_queue.cc:260] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:30.302928 16829 ts_tablet_manager.cc:1398] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:30.324290 16560 raft_consensus.cc:482] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:30.324431 16560 raft_consensus.cc:504] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:30.324854 16560 leader_election.cc:288] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237), 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:30.325332 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e76bd0ecd3e4c168e141fb7984af3ec" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" is_pre_election: true
I20240502 14:00:30.325332 15701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e76bd0ecd3e4c168e141fb7984af3ec" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917" is_pre_election: true
I20240502 14:00:30.325506 15571 raft_consensus.cc:2454] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 0.
I20240502 14:00:30.325506 15701 raft_consensus.cc:2454] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 0.
I20240502 14:00:30.325901 15899 leader_election.cc:302] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 4d0f8f311e834d81bd54b9f8e7923bcd, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:30.326081 16560 raft_consensus.cc:2781] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:30.326156 16560 raft_consensus.cc:482] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:30.326206 16560 raft_consensus.cc:3037] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.327106 16560 raft_consensus.cc:504] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:30.327447 16560 leader_election.cc:288] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 election: Requested vote from peers 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237), 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:30.327811 15701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e76bd0ecd3e4c168e141fb7984af3ec" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917"
I20240502 14:00:30.327834 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e76bd0ecd3e4c168e141fb7984af3ec" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
I20240502 14:00:30.327936 15571 raft_consensus.cc:3037] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.327936 15701 raft_consensus.cc:3037] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.328863 15571 raft_consensus.cc:2454] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 1.
I20240502 14:00:30.328864 15701 raft_consensus.cc:2454] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 1.
I20240502 14:00:30.329160 15899 leader_election.cc:302] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 4d0f8f311e834d81bd54b9f8e7923bcd; no voters:
I20240502 14:00:30.329388 16560 raft_consensus.cc:2781] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:30.329593 16560 raft_consensus.cc:686] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Becoming Leader. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Running, Role: LEADER
I20240502 14:00:30.329746 16560 consensus_queue.cc:237] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:30.331461 15420 catalog_manager.cc:5484] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132). New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:30.419346 16560 raft_consensus.cc:482] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:30.419469 16560 raft_consensus.cc:504] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:30.419903 16560 leader_election.cc:288] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531), 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:30.420322 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d92e5885bf94c3e8e0e4dee40017699" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf" is_pre_election: true
I20240502 14:00:30.420325 15701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d92e5885bf94c3e8e0e4dee40017699" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917" is_pre_election: true
I20240502 14:00:30.420460 15833 raft_consensus.cc:2454] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 0.
I20240502 14:00:30.420457 15701 raft_consensus.cc:2454] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 0.
I20240502 14:00:30.420749 15900 leader_election.cc:302] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:30.420962 16560 raft_consensus.cc:2781] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:30.421023 16560 raft_consensus.cc:482] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:30.421066 16560 raft_consensus.cc:3037] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.422022 16560 raft_consensus.cc:504] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:30.422335 16560 leader_election.cc:288] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 election: Requested vote from peers 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531), 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:30.422586 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d92e5885bf94c3e8e0e4dee40017699" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf"
I20240502 14:00:30.422654 15701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d92e5885bf94c3e8e0e4dee40017699" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917"
I20240502 14:00:30.422694 15833 raft_consensus.cc:3037] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.422765 15701 raft_consensus.cc:3037] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.423437 15833 raft_consensus.cc:2454] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 1.
I20240502 14:00:30.423441 15701 raft_consensus.cc:2454] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 1.
I20240502 14:00:30.423696 15897 leader_election.cc:302] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:30.423838 16560 raft_consensus.cc:2781] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:30.423920 16560 raft_consensus.cc:686] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Becoming Leader. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Running, Role: LEADER
I20240502 14:00:30.424057 16560 consensus_queue.cc:237] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:30.425849 15420 catalog_manager.cc:5484] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132). New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:30.440013 16780 raft_consensus.cc:482] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:30.440131 16780 raft_consensus.cc:504] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:30.440536 16780 leader_election.cc:288] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:30.440874 15701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb9f100600a246708457280d91826fce" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917" is_pre_election: true
I20240502 14:00:30.440984 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb9f100600a246708457280d91826fce" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf" is_pre_election: true
I20240502 14:00:30.441018 15701 raft_consensus.cc:2454] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 0.
I20240502 14:00:30.441108 15833 raft_consensus.cc:2454] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 0.
I20240502 14:00:30.441296 15504 leader_election.cc:302] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:30.441468 16780 raft_consensus.cc:2781] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:30.441530 16780 raft_consensus.cc:482] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:30.441566 16780 raft_consensus.cc:3037] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.442384 16780 raft_consensus.cc:504] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:30.442727 16780 leader_election.cc:288] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 election: Requested vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:30.443089 15701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb9f100600a246708457280d91826fce" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917"
I20240502 14:00:30.443125 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb9f100600a246708457280d91826fce" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf"
I20240502 14:00:30.443217 15833 raft_consensus.cc:3037] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.443214 15701 raft_consensus.cc:3037] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.444128 15833 raft_consensus.cc:2454] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 1.
I20240502 14:00:30.444126 15701 raft_consensus.cc:2454] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 1.
I20240502 14:00:30.444401 15507 leader_election.cc:302] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:30.444553 16780 raft_consensus.cc:2781] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:30.444638 16780 raft_consensus.cc:686] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Becoming Leader. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Running, Role: LEADER
I20240502 14:00:30.444779 16780 consensus_queue.cc:237] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:30.446434 15420 catalog_manager.cc:5484] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129). New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } }
W20240502 14:00:30.594106 16846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:30.594372 16846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:30.597479 16846 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 14:00:30.695910 15833 raft_consensus.cc:1261] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:30.695935 15702 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:30.695935 15701 raft_consensus.cc:1261] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:30.696077 15571 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:30.696358 16560 consensus_queue.cc:1035] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.696699 16560 consensus_queue.cc:1035] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.696956 16560 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.697238 16560 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.702833 15832 raft_consensus.cc:1261] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:30.704349 16753 consensus_queue.cc:1035] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.706569 15702 raft_consensus.cc:1261] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:30.706993 16753 consensus_queue.cc:1035] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 116.815 ms
time per row: 0.0584074 ms
I20240502 14:00:30.761529 15427 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49520:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:30.767076 15550 tablet_service.cc:1467] Processing CreateTablet for tablet 2886dfbec1e141198b0cf7a435fac498 (DEFAULT_TABLE table=rebalance_test_table_5 [id=0af7e16dce074d40ae6277d41e034322]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.767253 15812 tablet_service.cc:1467] Processing CreateTablet for tablet 2886dfbec1e141198b0cf7a435fac498 (DEFAULT_TABLE table=rebalance_test_table_5 [id=0af7e16dce074d40ae6277d41e034322]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.767403 15550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2886dfbec1e141198b0cf7a435fac498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.767534 15812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2886dfbec1e141198b0cf7a435fac498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.767680 15811 tablet_service.cc:1467] Processing CreateTablet for tablet 436c127af87c49e8b7bf9b6571f7dc30 (DEFAULT_TABLE table=rebalance_test_table_5 [id=0af7e16dce074d40ae6277d41e034322]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.767884 15811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436c127af87c49e8b7bf9b6571f7dc30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.768343 15944 tablet_service.cc:1467] Processing CreateTablet for tablet 2886dfbec1e141198b0cf7a435fac498 (DEFAULT_TABLE table=rebalance_test_table_5 [id=0af7e16dce074d40ae6277d41e034322]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.768623 15944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2886dfbec1e141198b0cf7a435fac498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.769552 16830 tablet_bootstrap.cc:492] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:30.769757 15813 tablet_service.cc:1467] Processing CreateTablet for tablet b0cd21dc897b41aca9118be6df48cc0c (DEFAULT_TABLE table=rebalance_test_table_5 [id=0af7e16dce074d40ae6277d41e034322]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.769956 15813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0cd21dc897b41aca9118be6df48cc0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.770535 16830 tablet_bootstrap.cc:654] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.770567 16834 tablet_bootstrap.cc:492] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:30.770748 15943 tablet_service.cc:1467] Processing CreateTablet for tablet b0cd21dc897b41aca9118be6df48cc0c (DEFAULT_TABLE table=rebalance_test_table_5 [id=0af7e16dce074d40ae6277d41e034322]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.770948 15943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0cd21dc897b41aca9118be6df48cc0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.771425 16834 tablet_bootstrap.cc:654] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.771955 15681 tablet_service.cc:1467] Processing CreateTablet for tablet 436c127af87c49e8b7bf9b6571f7dc30 (DEFAULT_TABLE table=rebalance_test_table_5 [id=0af7e16dce074d40ae6277d41e034322]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.772233 15681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436c127af87c49e8b7bf9b6571f7dc30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.772555 16834 tablet_bootstrap.cc:492] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:30.772626 16834 ts_tablet_manager.cc:1367] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:30.773084 16834 raft_consensus.cc:348] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:30.773190 16834 raft_consensus.cc:374] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.773224 16834 raft_consensus.cc:729] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.773325 16834 consensus_queue.cc:260] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:30.773605 16834 ts_tablet_manager.cc:1398] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:30.773759 16834 tablet_bootstrap.cc:492] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:30.773988 16829 tablet_bootstrap.cc:492] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:30.774061 16831 tablet_bootstrap.cc:492] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:30.774698 15551 tablet_service.cc:1467] Processing CreateTablet for tablet 436c127af87c49e8b7bf9b6571f7dc30 (DEFAULT_TABLE table=rebalance_test_table_5 [id=0af7e16dce074d40ae6277d41e034322]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.774824 16829 tablet_bootstrap.cc:654] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.774947 16831 tablet_bootstrap.cc:654] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.774971 15551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436c127af87c49e8b7bf9b6571f7dc30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.775952 16834 tablet_bootstrap.cc:654] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.776002 16831 tablet_bootstrap.cc:492] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:30.776079 16831 ts_tablet_manager.cc:1367] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:30.776540 16831 raft_consensus.cc:348] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:30.776661 16831 raft_consensus.cc:374] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.776700 16831 raft_consensus.cc:729] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.776824 16831 consensus_queue.cc:260] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:30.776968 16830 tablet_bootstrap.cc:492] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:30.777062 16830 ts_tablet_manager.cc:1367] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:30.777091 16831 ts_tablet_manager.cc:1398] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:30.777256 16831 tablet_bootstrap.cc:492] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:30.777556 16830 raft_consensus.cc:348] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:30.777679 16830 raft_consensus.cc:374] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.777724 16830 raft_consensus.cc:729] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.777853 16830 consensus_queue.cc:260] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:30.778038 16831 tablet_bootstrap.cc:654] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.778124 16830 ts_tablet_manager.cc:1398] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:30.778178 15550 tablet_service.cc:1467] Processing CreateTablet for tablet b0cd21dc897b41aca9118be6df48cc0c (DEFAULT_TABLE table=rebalance_test_table_5 [id=0af7e16dce074d40ae6277d41e034322]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.778472 16830 tablet_bootstrap.cc:492] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:30.779350 16830 tablet_bootstrap.cc:654] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.779599 16834 tablet_bootstrap.cc:492] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:30.779675 15550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0cd21dc897b41aca9118be6df48cc0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.779700 16834 ts_tablet_manager.cc:1367] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:00:30.780148 16834 raft_consensus.cc:348] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:30.780252 16834 raft_consensus.cc:374] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.780288 16834 raft_consensus.cc:729] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.780438 16834 consensus_queue.cc:260] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:30.780776 16834 ts_tablet_manager.cc:1398] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:30.781608 16830 tablet_bootstrap.cc:492] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:30.781701 16830 ts_tablet_manager.cc:1367] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:30.782042 16830 raft_consensus.cc:348] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:30.782124 16830 raft_consensus.cc:374] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.782156 16830 raft_consensus.cc:729] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.782334 16830 consensus_queue.cc:260] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:30.782572 16831 tablet_bootstrap.cc:492] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:30.782586 16830 ts_tablet_manager.cc:1398] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:30.782665 16831 ts_tablet_manager.cc:1367] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:30.782747 16830 tablet_bootstrap.cc:492] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:30.783087 16831 raft_consensus.cc:348] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:30.783174 16831 raft_consensus.cc:374] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.783196 16831 raft_consensus.cc:729] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.783298 16831 consensus_queue.cc:260] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:30.783552 16831 ts_tablet_manager.cc:1398] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:30.783703 16831 tablet_bootstrap.cc:492] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:30.783802 16830 tablet_bootstrap.cc:654] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.784523 16831 tablet_bootstrap.cc:654] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.784920 16830 tablet_bootstrap.cc:492] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:30.785013 16830 ts_tablet_manager.cc:1367] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:30.785470 16830 raft_consensus.cc:348] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:30.785568 16830 raft_consensus.cc:374] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.785738 16830 raft_consensus.cc:729] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.785856 16830 consensus_queue.cc:260] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:30.786196 16830 ts_tablet_manager.cc:1398] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:30.787097 16829 tablet_bootstrap.cc:492] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:30.787192 16831 tablet_bootstrap.cc:492] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:30.787202 16829 ts_tablet_manager.cc:1367] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240502 14:00:30.787277 16831 ts_tablet_manager.cc:1367] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:00:30.787628 16829 raft_consensus.cc:348] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:30.787676 16831 raft_consensus.cc:348] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:30.787744 16829 raft_consensus.cc:374] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.787778 16829 raft_consensus.cc:729] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.787779 16831 raft_consensus.cc:374] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.787819 16831 raft_consensus.cc:729] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.787912 16829 consensus_queue.cc:260] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:30.787938 16831 consensus_queue.cc:260] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:30.788182 16829 ts_tablet_manager.cc:1398] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:30.788190 16831 ts_tablet_manager.cc:1398] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:30.852978 16780 raft_consensus.cc:482] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:30.853112 16780 raft_consensus.cc:504] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:30.853547 16780 leader_election.cc:288] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719), 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:30.853935 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0cd21dc897b41aca9118be6df48cc0c" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" is_pre_election: true
I20240502 14:00:30.853993 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0cd21dc897b41aca9118be6df48cc0c" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf" is_pre_election: true
I20240502 14:00:30.854086 15964 raft_consensus.cc:2454] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 0.
I20240502 14:00:30.854161 15833 raft_consensus.cc:2454] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 0.
I20240502 14:00:30.854437 15504 leader_election.cc:302] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 4d0f8f311e834d81bd54b9f8e7923bcd; no voters:
I20240502 14:00:30.854635 16780 raft_consensus.cc:2781] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:30.854707 16780 raft_consensus.cc:482] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:30.854740 16780 raft_consensus.cc:3037] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.855713 16780 raft_consensus.cc:504] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:30.856029 16780 leader_election.cc:288] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 election: Requested vote from peers 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719), 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:30.856339 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0cd21dc897b41aca9118be6df48cc0c" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
I20240502 14:00:30.856377 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0cd21dc897b41aca9118be6df48cc0c" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf"
I20240502 14:00:30.856465 15833 raft_consensus.cc:3037] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.856464 15964 raft_consensus.cc:3037] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.857287 15964 raft_consensus.cc:2454] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 1.
I20240502 14:00:30.857347 15833 raft_consensus.cc:2454] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 1.
I20240502 14:00:30.857571 15504 leader_election.cc:302] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 4d0f8f311e834d81bd54b9f8e7923bcd; no voters:
I20240502 14:00:30.857781 16780 raft_consensus.cc:2781] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:30.857860 16780 raft_consensus.cc:686] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Becoming Leader. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Running, Role: LEADER
I20240502 14:00:30.858004 16780 consensus_queue.cc:237] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:30.859798 15425 catalog_manager.cc:5484] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129). New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } }
I20240502 14:00:30.860309 16780 raft_consensus.cc:482] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:30.860395 16780 raft_consensus.cc:504] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:30.860720 16780 leader_election.cc:288] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531), 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:30.861150 15700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436c127af87c49e8b7bf9b6571f7dc30" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917" is_pre_election: true
I20240502 14:00:30.861279 15700 raft_consensus.cc:2454] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 0.
I20240502 14:00:30.861541 15504 leader_election.cc:302] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:30.861676 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436c127af87c49e8b7bf9b6571f7dc30" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf" is_pre_election: true
I20240502 14:00:30.861768 15833 raft_consensus.cc:2454] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 0.
I20240502 14:00:30.861927 16780 raft_consensus.cc:2781] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:30.861977 16780 raft_consensus.cc:482] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:30.862017 16780 raft_consensus.cc:3037] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.879119 16780 raft_consensus.cc:504] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:30.879518 16780 leader_election.cc:288] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 election: Requested vote from peers 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531), 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:30.879913 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436c127af87c49e8b7bf9b6571f7dc30" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf"
I20240502 14:00:30.879937 15700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436c127af87c49e8b7bf9b6571f7dc30" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917"
I20240502 14:00:30.880028 15833 raft_consensus.cc:3037] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.880076 15700 raft_consensus.cc:3037] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.880781 15833 raft_consensus.cc:2454] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 1.
I20240502 14:00:30.880877 15700 raft_consensus.cc:2454] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 1.
I20240502 14:00:30.881153 15507 leader_election.cc:302] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:30.881351 16780 raft_consensus.cc:2781] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:30.881446 16780 raft_consensus.cc:686] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Becoming Leader. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Running, Role: LEADER
I20240502 14:00:30.881582 16826 raft_consensus.cc:482] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:30.881611 16780 consensus_queue.cc:237] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:30.881680 16826 raft_consensus.cc:504] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:30.882081 16826 leader_election.cc:288] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237), 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:30.882373 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2886dfbec1e141198b0cf7a435fac498" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" is_pre_election: true
I20240502 14:00:30.882494 15571 raft_consensus.cc:2454] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 0.
I20240502 14:00:30.882483 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2886dfbec1e141198b0cf7a435fac498" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" is_pre_election: true
I20240502 14:00:30.882637 15964 raft_consensus.cc:2454] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 0.
I20240502 14:00:30.882918 15766 leader_election.cc:302] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:30.883154 16826 raft_consensus.cc:2781] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:30.883219 16826 raft_consensus.cc:482] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:30.883265 16826 raft_consensus.cc:3037] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.883625 15425 catalog_manager.cc:5484] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129). New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } }
I20240502 14:00:30.884224 16826 raft_consensus.cc:504] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:30.884533 16826 leader_election.cc:288] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 election: Requested vote from peers 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237), 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:30.884820 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2886dfbec1e141198b0cf7a435fac498" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
I20240502 14:00:30.884904 15571 raft_consensus.cc:3037] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.884884 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2886dfbec1e141198b0cf7a435fac498" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
I20240502 14:00:30.884989 15964 raft_consensus.cc:3037] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.885646 15571 raft_consensus.cc:2454] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 1.
I20240502 14:00:30.885910 15768 leader_election.cc:302] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:30.885982 15964 raft_consensus.cc:2454] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 1.
I20240502 14:00:30.886049 16826 raft_consensus.cc:2781] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:30.886200 16826 raft_consensus.cc:686] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Becoming Leader. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Running, Role: LEADER
I20240502 14:00:30.886346 16826 consensus_queue.cc:237] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:30.887837 15425 catalog_manager.cc:5484] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131). New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } }
W20240502 14:00:31.009176 16901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:31.009398 16901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:31.012377 16901 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 14:00:31.091699 15833 raft_consensus.cc:1261] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:31.092129 16780 consensus_queue.cc:1035] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.096410 15964 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:31.097885 15700 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:31.098351 16753 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.098415 15571 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:31.098623 15964 raft_consensus.cc:1261] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:31.098949 16753 consensus_queue.cc:1035] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.098999 16826 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.099152 15832 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:31.099267 16826 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.103652 16753 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.127560 16777 mvcc.cc:205] Tried to move back new op lower bound from 7023240933797175296 to 7023240932892442624. Current Snapshot: MvccSnapshot[applied={T|T < 7023240933748301824}]
I20240502 14:00:31.133430 16876 mvcc.cc:205] Tried to move back new op lower bound from 7023240933748101120 to 7023240932795662336. Current Snapshot: MvccSnapshot[applied={T|T < 7023240933748101120}]
I20240502 14:00:31.139870 16779 mvcc.cc:205] Tried to move back new op lower bound from 7023240933762994176 to 7023240932911890432. Current Snapshot: MvccSnapshot[applied={T|T < 7023240933762994176 or (T in {7023240933762994176})}]
INSERT report
rows total: 2000
time total: 102.105 ms
time per row: 0.0510527 ms
I20240502 14:00:31.159070 15428 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49520:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:31.164368 15550 tablet_service.cc:1467] Processing CreateTablet for tablet dfdcf1124e8a474facc14c45e8963568 (DEFAULT_TABLE table=rebalance_test_table_6 [id=6dcd44a470ab434a988eea52c3916f26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.164476 15681 tablet_service.cc:1467] Processing CreateTablet for tablet dfdcf1124e8a474facc14c45e8963568 (DEFAULT_TABLE table=rebalance_test_table_6 [id=6dcd44a470ab434a988eea52c3916f26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.164685 15550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfdcf1124e8a474facc14c45e8963568. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.164729 15681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfdcf1124e8a474facc14c45e8963568. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.165419 15551 tablet_service.cc:1467] Processing CreateTablet for tablet 837e8b1235a44131bdaef24b763d1cac (DEFAULT_TABLE table=rebalance_test_table_6 [id=6dcd44a470ab434a988eea52c3916f26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.165387 15944 tablet_service.cc:1467] Processing CreateTablet for tablet 22eef55211ca4f65979022d2f4ae829f (DEFAULT_TABLE table=rebalance_test_table_6 [id=6dcd44a470ab434a988eea52c3916f26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.165396 15942 tablet_service.cc:1467] Processing CreateTablet for tablet 837e8b1235a44131bdaef24b763d1cac (DEFAULT_TABLE table=rebalance_test_table_6 [id=6dcd44a470ab434a988eea52c3916f26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.165657 15551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837e8b1235a44131bdaef24b763d1cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.165726 15944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22eef55211ca4f65979022d2f4ae829f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.165902 15942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837e8b1235a44131bdaef24b763d1cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.166133 15943 tablet_service.cc:1467] Processing CreateTablet for tablet dfdcf1124e8a474facc14c45e8963568 (DEFAULT_TABLE table=rebalance_test_table_6 [id=6dcd44a470ab434a988eea52c3916f26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.166378 15943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfdcf1124e8a474facc14c45e8963568. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.166810 16830 tablet_bootstrap.cc:492] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:31.166921 15682 tablet_service.cc:1467] Processing CreateTablet for tablet 22eef55211ca4f65979022d2f4ae829f (DEFAULT_TABLE table=rebalance_test_table_6 [id=6dcd44a470ab434a988eea52c3916f26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.167119 15682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22eef55211ca4f65979022d2f4ae829f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.167526 15680 tablet_service.cc:1467] Processing CreateTablet for tablet 837e8b1235a44131bdaef24b763d1cac (DEFAULT_TABLE table=rebalance_test_table_6 [id=6dcd44a470ab434a988eea52c3916f26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.167654 16830 tablet_bootstrap.cc:654] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.167773 15680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837e8b1235a44131bdaef24b763d1cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.167897 16834 tablet_bootstrap.cc:492] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:31.168751 16834 tablet_bootstrap.cc:654] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.170078 16830 tablet_bootstrap.cc:492] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:31.170015 15813 tablet_service.cc:1467] Processing CreateTablet for tablet 22eef55211ca4f65979022d2f4ae829f (DEFAULT_TABLE table=rebalance_test_table_6 [id=6dcd44a470ab434a988eea52c3916f26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.170164 16830 ts_tablet_manager.cc:1367] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:31.170280 15813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22eef55211ca4f65979022d2f4ae829f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.170272 16834 tablet_bootstrap.cc:492] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:31.170392 16834 ts_tablet_manager.cc:1367] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:31.170670 16830 raft_consensus.cc:348] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.170751 16830 raft_consensus.cc:374] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.170771 16830 raft_consensus.cc:729] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.170819 16834 raft_consensus.cc:348] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:31.170881 16830 consensus_queue.cc:260] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.170924 16834 raft_consensus.cc:374] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.170966 16834 raft_consensus.cc:729] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.171070 16834 consensus_queue.cc:260] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:31.171156 16830 ts_tablet_manager.cc:1398] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:31.171303 16830 tablet_bootstrap.cc:492] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:31.171357 16834 ts_tablet_manager.cc:1398] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:31.171506 16834 tablet_bootstrap.cc:492] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:31.172088 16830 tablet_bootstrap.cc:654] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.172461 16831 tablet_bootstrap.cc:492] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:31.172569 16834 tablet_bootstrap.cc:654] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.173297 16831 tablet_bootstrap.cc:654] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.174366 16831 tablet_bootstrap.cc:492] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:31.174449 16831 ts_tablet_manager.cc:1367] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:31.174532 16830 tablet_bootstrap.cc:492] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:31.174597 16830 ts_tablet_manager.cc:1367] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:00:31.174875 16831 raft_consensus.cc:348] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:31.174997 16831 raft_consensus.cc:374] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.175006 16834 tablet_bootstrap.cc:492] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:31.175029 16831 raft_consensus.cc:729] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.175076 16834 ts_tablet_manager.cc:1367] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:00:31.175138 16831 consensus_queue.cc:260] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:31.175325 16830 raft_consensus.cc:348] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:31.175408 16830 raft_consensus.cc:374] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.175419 16831 ts_tablet_manager.cc:1398] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:31.175441 16830 raft_consensus.cc:729] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.175419 16834 raft_consensus.cc:348] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.175493 16834 raft_consensus.cc:374] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.175514 16834 raft_consensus.cc:729] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.175561 16830 consensus_queue.cc:260] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:31.175647 16834 consensus_queue.cc:260] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.175865 16830 ts_tablet_manager.cc:1398] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:31.175966 16834 ts_tablet_manager.cc:1398] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:31.177098 16834 tablet_bootstrap.cc:492] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:31.177994 16834 tablet_bootstrap.cc:654] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.180136 16829 tablet_bootstrap.cc:492] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:31.181090 16829 tablet_bootstrap.cc:654] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.182605 16829 tablet_bootstrap.cc:492] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:31.182698 16829 ts_tablet_manager.cc:1367] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:31.183142 16829 raft_consensus.cc:348] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.183250 16829 raft_consensus.cc:374] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.183293 16829 raft_consensus.cc:729] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.183435 16829 consensus_queue.cc:260] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.184401 16834 tablet_bootstrap.cc:492] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:31.184469 16834 ts_tablet_manager.cc:1367] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240502 14:00:31.184821 16834 raft_consensus.cc:348] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:31.184931 16834 raft_consensus.cc:374] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.184960 16834 raft_consensus.cc:729] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.185079 16834 consensus_queue.cc:260] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:31.185314 16834 ts_tablet_manager.cc:1398] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:31.185537 16829 ts_tablet_manager.cc:1398] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:31.185717 16829 tablet_bootstrap.cc:492] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:31.186559 16829 tablet_bootstrap.cc:654] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.187606 16829 tablet_bootstrap.cc:492] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:31.187683 16829 ts_tablet_manager.cc:1367] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:31.188027 16829 raft_consensus.cc:348] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:31.188143 16829 raft_consensus.cc:374] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.188199 16829 raft_consensus.cc:729] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.188320 16829 consensus_queue.cc:260] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:31.188570 16829 ts_tablet_manager.cc:1398] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:31.188823 16829 tablet_bootstrap.cc:492] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:31.189692 16829 tablet_bootstrap.cc:654] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.190834 16829 tablet_bootstrap.cc:492] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:31.190919 16829 ts_tablet_manager.cc:1367] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:31.191413 16829 raft_consensus.cc:348] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:31.191533 16829 raft_consensus.cc:374] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.191576 16829 raft_consensus.cc:729] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.191686 16829 consensus_queue.cc:260] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:31.191922 16829 ts_tablet_manager.cc:1398] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:31.264218 16899 raft_consensus.cc:482] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:31.264339 16899 raft_consensus.cc:504] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:31.264729 16899 leader_election.cc:288] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719), 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:31.265125 15963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22eef55211ca4f65979022d2f4ae829f" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" is_pre_election: true
I20240502 14:00:31.265201 15700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22eef55211ca4f65979022d2f4ae829f" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917" is_pre_election: true
I20240502 14:00:31.265287 15963 raft_consensus.cc:2454] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 0.
I20240502 14:00:31.265332 15700 raft_consensus.cc:2454] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 0.
I20240502 14:00:31.265612 15766 leader_election.cc:302] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:31.265794 16899 raft_consensus.cc:2781] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:31.265858 16899 raft_consensus.cc:482] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:31.265910 16899 raft_consensus.cc:3037] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.266845 16899 raft_consensus.cc:504] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:31.267153 16899 leader_election.cc:288] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 election: Requested vote from peers 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719), 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:31.267493 15963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22eef55211ca4f65979022d2f4ae829f" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
I20240502 14:00:31.267575 15700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22eef55211ca4f65979022d2f4ae829f" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917"
I20240502 14:00:31.267633 15963 raft_consensus.cc:3037] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.267709 15700 raft_consensus.cc:3037] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.268656 15700 raft_consensus.cc:2454] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 1.
I20240502 14:00:31.268654 15963 raft_consensus.cc:2454] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 1.
I20240502 14:00:31.269001 15766 leader_election.cc:302] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:31.269290 16899 raft_consensus.cc:2781] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:31.269381 16899 raft_consensus.cc:686] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Becoming Leader. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Running, Role: LEADER
I20240502 14:00:31.269536 16899 consensus_queue.cc:237] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:31.271595 15428 catalog_manager.cc:5484] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131). New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } }
I20240502 14:00:31.309921 16780 raft_consensus.cc:482] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:31.310079 16780 raft_consensus.cc:504] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.310714 16780 leader_election.cc:288] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:31.311038 15700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfdcf1124e8a474facc14c45e8963568" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917" is_pre_election: true
I20240502 14:00:31.311066 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfdcf1124e8a474facc14c45e8963568" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" is_pre_election: true
I20240502 14:00:31.311178 15700 raft_consensus.cc:2454] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 0.
I20240502 14:00:31.311210 15964 raft_consensus.cc:2454] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 0.
I20240502 14:00:31.311522 15504 leader_election.cc:302] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:31.311749 16780 raft_consensus.cc:2781] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:31.311820 16780 raft_consensus.cc:482] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:31.311849 16780 raft_consensus.cc:3037] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.312681 16780 raft_consensus.cc:504] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.313030 16780 leader_election.cc:288] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 election: Requested vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:31.313453 15700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfdcf1124e8a474facc14c45e8963568" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917"
I20240502 14:00:31.313475 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfdcf1124e8a474facc14c45e8963568" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
I20240502 14:00:31.313578 15964 raft_consensus.cc:3037] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.313592 15700 raft_consensus.cc:3037] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.314419 15964 raft_consensus.cc:2454] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 1.
I20240502 14:00:31.314420 15700 raft_consensus.cc:2454] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 1.
I20240502 14:00:31.314749 15504 leader_election.cc:302] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:31.314908 16780 raft_consensus.cc:2781] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:31.315003 16780 raft_consensus.cc:686] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Becoming Leader. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Running, Role: LEADER
I20240502 14:00:31.315177 16780 consensus_queue.cc:237] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.316972 15428 catalog_manager.cc:5484] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129). New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:31.349067 16877 raft_consensus.cc:482] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:31.349207 16877 raft_consensus.cc:504] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:31.349675 16877 leader_election.cc:288] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:31.350165 15700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837e8b1235a44131bdaef24b763d1cac" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917" is_pre_election: true
I20240502 14:00:31.350200 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837e8b1235a44131bdaef24b763d1cac" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" is_pre_election: true
I20240502 14:00:31.350356 15700 raft_consensus.cc:2454] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 0.
I20240502 14:00:31.350356 15571 raft_consensus.cc:2454] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 0.
I20240502 14:00:31.350714 15899 leader_election.cc:302] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 4d0f8f311e834d81bd54b9f8e7923bcd; no voters:
I20240502 14:00:31.350906 16877 raft_consensus.cc:2781] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:31.350972 16877 raft_consensus.cc:482] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:31.350999 16877 raft_consensus.cc:3037] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.351845 16877 raft_consensus.cc:504] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:31.352182 16877 leader_election.cc:288] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 election: Requested vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:31.352492 15700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837e8b1235a44131bdaef24b763d1cac" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917"
I20240502 14:00:31.352594 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837e8b1235a44131bdaef24b763d1cac" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
I20240502 14:00:31.352639 15700 raft_consensus.cc:3037] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.352705 15571 raft_consensus.cc:3037] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.353627 15700 raft_consensus.cc:2454] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 1.
I20240502 14:00:31.353614 15571 raft_consensus.cc:2454] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 1.
I20240502 14:00:31.353897 15897 leader_election.cc:302] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:31.354154 16877 raft_consensus.cc:2781] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:31.354362 16877 raft_consensus.cc:686] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Becoming Leader. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Running, Role: LEADER
I20240502 14:00:31.354583 16877 consensus_queue.cc:237] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } }
I20240502 14:00:31.356412 15428 catalog_manager.cc:5484] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132). New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } }
W20240502 14:00:31.469669 16948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:31.469906 16948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:31.472931 16948 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 14:00:31.556785 15571 raft_consensus.cc:1261] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:31.557425 16877 consensus_queue.cc:1035] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.561760 15964 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:31.562095 15700 raft_consensus.cc:1261] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:31.562412 15701 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:31.562636 16946 consensus_queue.cc:1035] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.562883 16826 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.563253 16826 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.567299 15963 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:31.567728 16882 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.574229 15701 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:31.577313 16882 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:31.591571 16816 mvcc.cc:205] Tried to move back new op lower bound from 7023240935675695104 to 7023240934830096384. Current Snapshot: MvccSnapshot[applied={T|T < 7023240935650869248}]
INSERT report
rows total: 2000
time total: 108.808 ms
time per row: 0.054404 ms
I20240502 14:00:31.626964 15420 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49520:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:31.632412 15813 tablet_service.cc:1467] Processing CreateTablet for tablet 76f07464d9bd49a9b1b9d58f6a180cf8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fc947d3d4f264291a81e7e95cb84f48a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.632537 15551 tablet_service.cc:1467] Processing CreateTablet for tablet 76f07464d9bd49a9b1b9d58f6a180cf8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fc947d3d4f264291a81e7e95cb84f48a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.632745 15813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f07464d9bd49a9b1b9d58f6a180cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.632843 15551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f07464d9bd49a9b1b9d58f6a180cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.632970 15812 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5e33f337bd47b2a6b11b29a5eacad8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fc947d3d4f264291a81e7e95cb84f48a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.633177 15812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5e33f337bd47b2a6b11b29a5eacad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.634583 15811 tablet_service.cc:1467] Processing CreateTablet for tablet 03dd735113ab4ea6bdb696b11a4efd7f (DEFAULT_TABLE table=rebalance_test_table_7 [id=fc947d3d4f264291a81e7e95cb84f48a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.634667 16830 tablet_bootstrap.cc:492] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:31.634816 15811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03dd735113ab4ea6bdb696b11a4efd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.635506 16830 tablet_bootstrap.cc:654] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.635802 16831 tablet_bootstrap.cc:492] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:31.636045 15943 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5e33f337bd47b2a6b11b29a5eacad8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fc947d3d4f264291a81e7e95cb84f48a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.636314 15943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5e33f337bd47b2a6b11b29a5eacad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.636310 15682 tablet_service.cc:1467] Processing CreateTablet for tablet 76f07464d9bd49a9b1b9d58f6a180cf8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fc947d3d4f264291a81e7e95cb84f48a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.636310 15680 tablet_service.cc:1467] Processing CreateTablet for tablet 03dd735113ab4ea6bdb696b11a4efd7f (DEFAULT_TABLE table=rebalance_test_table_7 [id=fc947d3d4f264291a81e7e95cb84f48a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.636613 15682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f07464d9bd49a9b1b9d58f6a180cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.636647 16831 tablet_bootstrap.cc:654] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.636778 15680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03dd735113ab4ea6bdb696b11a4efd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.638386 15551 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5e33f337bd47b2a6b11b29a5eacad8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fc947d3d4f264291a81e7e95cb84f48a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.638628 15551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5e33f337bd47b2a6b11b29a5eacad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.638633 16834 tablet_bootstrap.cc:492] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:31.639464 16834 tablet_bootstrap.cc:654] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.640105 16830 tablet_bootstrap.cc:492] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:31.640130 16829 tablet_bootstrap.cc:492] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:31.640187 16830 ts_tablet_manager.cc:1367] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240502 14:00:31.640331 16831 tablet_bootstrap.cc:492] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:31.640404 16831 ts_tablet_manager.cc:1367] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:31.640668 16830 raft_consensus.cc:348] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:31.640774 16830 raft_consensus.cc:374] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.640805 16830 raft_consensus.cc:729] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.640890 16829 tablet_bootstrap.cc:654] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.640849 16831 raft_consensus.cc:348] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:31.640944 16831 raft_consensus.cc:374] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.640977 16831 raft_consensus.cc:729] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.640944 16830 consensus_queue.cc:260] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:31.641096 16831 consensus_queue.cc:260] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:31.641243 16830 ts_tablet_manager.cc:1398] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:31.641356 16831 ts_tablet_manager.cc:1398] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:31.641391 16830 tablet_bootstrap.cc:492] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:31.641503 16831 tablet_bootstrap.cc:492] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:31.642432 15944 tablet_service.cc:1467] Processing CreateTablet for tablet 03dd735113ab4ea6bdb696b11a4efd7f (DEFAULT_TABLE table=rebalance_test_table_7 [id=fc947d3d4f264291a81e7e95cb84f48a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:31.642653 16830 tablet_bootstrap.cc:654] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.642699 15944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03dd735113ab4ea6bdb696b11a4efd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.643787 16830 tablet_bootstrap.cc:492] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:31.643864 16830 ts_tablet_manager.cc:1367] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:31.644330 16830 raft_consensus.cc:348] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.644426 16830 raft_consensus.cc:374] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.644456 16830 raft_consensus.cc:729] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.644587 16830 consensus_queue.cc:260] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.644762 16831 tablet_bootstrap.cc:654] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.645426 16830 ts_tablet_manager.cc:1398] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:31.646106 16831 tablet_bootstrap.cc:492] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:31.646111 16829 tablet_bootstrap.cc:492] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:31.646188 16831 ts_tablet_manager.cc:1367] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:31.646189 16829 ts_tablet_manager.cc:1367] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:31.646664 16831 raft_consensus.cc:348] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.646690 16829 raft_consensus.cc:348] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:31.646770 16831 raft_consensus.cc:374] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.646780 16829 raft_consensus.cc:374] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.646806 16831 raft_consensus.cc:729] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.646813 16829 raft_consensus.cc:729] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.646935 16831 consensus_queue.cc:260] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.646935 16829 consensus_queue.cc:260] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:31.647200 16831 ts_tablet_manager.cc:1398] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:31.647204 16829 ts_tablet_manager.cc:1398] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:31.647343 16829 tablet_bootstrap.cc:492] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:31.647343 16831 tablet_bootstrap.cc:492] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:31.648229 16831 tablet_bootstrap.cc:654] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.649226 16829 tablet_bootstrap.cc:654] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.650110 16831 tablet_bootstrap.cc:492] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:31.650192 16831 ts_tablet_manager.cc:1367] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:00:31.650458 16834 tablet_bootstrap.cc:492] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:31.650559 16834 ts_tablet_manager.cc:1367] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.001s
I20240502 14:00:31.650750 16831 raft_consensus.cc:348] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.650841 16831 raft_consensus.cc:374] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.650873 16831 raft_consensus.cc:729] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.650975 16831 consensus_queue.cc:260] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.651044 16834 raft_consensus.cc:348] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.651152 16834 raft_consensus.cc:374] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.651182 16834 raft_consensus.cc:729] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.651291 16831 ts_tablet_manager.cc:1398] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:31.651309 16834 consensus_queue.cc:260] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.651595 16834 ts_tablet_manager.cc:1398] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:31.651762 16834 tablet_bootstrap.cc:492] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:31.654067 16834 tablet_bootstrap.cc:654] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.655014 16829 tablet_bootstrap.cc:492] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:31.655129 16829 ts_tablet_manager.cc:1367] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:31.655360 16834 tablet_bootstrap.cc:492] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:31.655421 16834 ts_tablet_manager.cc:1367] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:00:31.655635 16829 raft_consensus.cc:348] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.655750 16829 raft_consensus.cc:374] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.655793 16829 raft_consensus.cc:729] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.655875 16834 raft_consensus.cc:348] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.655977 16834 raft_consensus.cc:374] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.656001 16834 raft_consensus.cc:729] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.655939 16829 consensus_queue.cc:260] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.656080 16834 consensus_queue.cc:260] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.656244 16829 ts_tablet_manager.cc:1398] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:31.656303 16834 ts_tablet_manager.cc:1398] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:31.709878 16946 raft_consensus.cc:482] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:31.710021 16946 raft_consensus.cc:504] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.710479 16946 leader_election.cc:288] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531), 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:31.710848 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5e33f337bd47b2a6b11b29a5eacad8" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" is_pre_election: true
I20240502 14:00:31.710865 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5e33f337bd47b2a6b11b29a5eacad8" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf" is_pre_election: true
I20240502 14:00:31.710999 15833 raft_consensus.cc:2454] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 0.
I20240502 14:00:31.710999 15571 raft_consensus.cc:2454] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 0.
I20240502 14:00:31.711295 15900 leader_election.cc:302] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:31.711498 16946 raft_consensus.cc:2781] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:31.711553 16946 raft_consensus.cc:482] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:31.711581 16946 raft_consensus.cc:3037] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.712509 16946 raft_consensus.cc:504] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.712875 16946 leader_election.cc:288] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 election: Requested vote from peers 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531), 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:31.713256 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5e33f337bd47b2a6b11b29a5eacad8" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf"
I20240502 14:00:31.713259 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5e33f337bd47b2a6b11b29a5eacad8" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
I20240502 14:00:31.713399 15571 raft_consensus.cc:3037] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.713399 15833 raft_consensus.cc:3037] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.714419 15833 raft_consensus.cc:2454] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 1.
I20240502 14:00:31.714419 15571 raft_consensus.cc:2454] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 1.
I20240502 14:00:31.714726 15899 leader_election.cc:302] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 4d0f8f311e834d81bd54b9f8e7923bcd; no voters:
I20240502 14:00:31.714888 16946 raft_consensus.cc:2781] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:31.715013 16946 raft_consensus.cc:686] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Becoming Leader. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Running, Role: LEADER
I20240502 14:00:31.715157 16946 consensus_queue.cc:237] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.717087 15420 catalog_manager.cc:5484] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132). New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } }
I20240502 14:00:31.820058 16826 raft_consensus.cc:482] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:31.820206 16826 raft_consensus.cc:504] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.820618 16826 leader_election.cc:288] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:31.820986 16946 raft_consensus.cc:482] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:31.821120 16946 raft_consensus.cc:504] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.821199 15963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03dd735113ab4ea6bdb696b11a4efd7f" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" is_pre_election: true
I20240502 14:00:31.821153 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03dd735113ab4ea6bdb696b11a4efd7f" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917" is_pre_election: true
I20240502 14:00:31.821285 15963 raft_consensus.cc:2454] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 0.
I20240502 14:00:31.821305 15702 raft_consensus.cc:2454] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 0.
I20240502 14:00:31.821519 16946 leader_election.cc:288] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531), 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:31.821609 15766 leader_election.cc:302] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:31.821801 16826 raft_consensus.cc:2781] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:31.821856 16826 raft_consensus.cc:482] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:31.821895 16826 raft_consensus.cc:3037] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.821954 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03dd735113ab4ea6bdb696b11a4efd7f" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf" is_pre_election: true
I20240502 14:00:31.822001 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03dd735113ab4ea6bdb696b11a4efd7f" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917" is_pre_election: true
I20240502 14:00:31.822104 15702 raft_consensus.cc:2454] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 0.
I20240502 14:00:31.822391 15897 leader_election.cc:302] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:31.822571 16946 raft_consensus.cc:2781] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:31.822623 16946 raft_consensus.cc:482] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:31.822657 16946 raft_consensus.cc:3037] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.823060 16826 raft_consensus.cc:504] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.823297 15833 raft_consensus.cc:2379] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d0f8f311e834d81bd54b9f8e7923bcd in current term 1: Already voted for candidate 82d7b7201cfc431d90baca1ad0a397cf in this term.
I20240502 14:00:31.823415 16826 leader_election.cc:288] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 election: Requested vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:31.823679 16946 raft_consensus.cc:504] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.823864 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03dd735113ab4ea6bdb696b11a4efd7f" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917"
I20240502 14:00:31.823968 15702 raft_consensus.cc:3037] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.824004 15963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03dd735113ab4ea6bdb696b11a4efd7f" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
I20240502 14:00:31.824159 15963 raft_consensus.cc:2379] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 82d7b7201cfc431d90baca1ad0a397cf in current term 1: Already voted for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in this term.
I20240502 14:00:31.824205 16946 leader_election.cc:288] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 election: Requested vote from peers 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531), 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:31.824487 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03dd735113ab4ea6bdb696b11a4efd7f" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf"
I20240502 14:00:31.824580 15833 raft_consensus.cc:2379] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d0f8f311e834d81bd54b9f8e7923bcd in current term 1: Already voted for candidate 82d7b7201cfc431d90baca1ad0a397cf in this term.
I20240502 14:00:31.824805 15700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03dd735113ab4ea6bdb696b11a4efd7f" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917"
I20240502 14:00:31.824894 15702 raft_consensus.cc:2454] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 1.
I20240502 14:00:31.825166 15897 leader_election.cc:302] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 82d7b7201cfc431d90baca1ad0a397cf, 8484181c4ece483c8b408817fab1a917
I20240502 14:00:31.825212 15766 leader_election.cc:302] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82d7b7201cfc431d90baca1ad0a397cf, 8484181c4ece483c8b408817fab1a917; no voters: 4d0f8f311e834d81bd54b9f8e7923bcd
I20240502 14:00:31.825352 16826 raft_consensus.cc:2781] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:31.825436 16826 raft_consensus.cc:686] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Becoming Leader. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Running, Role: LEADER
I20240502 14:00:31.825424 16946 raft_consensus.cc:2726] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:00:31.825588 16826 consensus_queue.cc:237] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:31.827376 15420 catalog_manager.cc:5484] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131). New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:31.831821 16984 raft_consensus.cc:482] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:31.831920 16984 raft_consensus.cc:504] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:31.832260 16984 leader_election.cc:288] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531), 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:31.832643 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f07464d9bd49a9b1b9d58f6a180cf8" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf" is_pre_election: true
I20240502 14:00:31.832645 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f07464d9bd49a9b1b9d58f6a180cf8" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" is_pre_election: true
I20240502 14:00:31.832775 15833 raft_consensus.cc:2454] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 0.
I20240502 14:00:31.832775 15571 raft_consensus.cc:2454] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 0.
I20240502 14:00:31.833010 15638 leader_election.cc:302] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82d7b7201cfc431d90baca1ad0a397cf, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:31.833184 16984 raft_consensus.cc:2781] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:31.833267 16984 raft_consensus.cc:482] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:31.833295 16984 raft_consensus.cc:3037] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.834086 16984 raft_consensus.cc:504] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:31.834421 16984 leader_election.cc:288] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 election: Requested vote from peers 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531), 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:31.834736 15571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f07464d9bd49a9b1b9d58f6a180cf8" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
I20240502 14:00:31.834743 15833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76f07464d9bd49a9b1b9d58f6a180cf8" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf"
I20240502 14:00:31.834836 15833 raft_consensus.cc:3037] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.834844 15571 raft_consensus.cc:3037] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.835520 15571 raft_consensus.cc:2454] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 1.
I20240502 14:00:31.835527 15833 raft_consensus.cc:2454] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 1.
I20240502 14:00:31.835779 15638 leader_election.cc:302] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82d7b7201cfc431d90baca1ad0a397cf, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:31.835933 16984 raft_consensus.cc:2781] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:31.836032 16984 raft_consensus.cc:686] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Becoming Leader. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Running, Role: LEADER
I20240502 14:00:31.836189 16984 consensus_queue.cc:237] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:31.837886 15420 catalog_manager.cc:5484] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8484181c4ece483c8b408817fab1a917 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } }
W20240502 14:00:31.983404 17005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:31.983650 17005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:31.986696 17005 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 14:00:32.062568 15963 raft_consensus.cc:1261] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.063704 15700 raft_consensus.cc:1261] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.064267 16899 consensus_queue.cc:1035] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.064596 16899 consensus_queue.cc:1035] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.065275 15571 raft_consensus.cc:1261] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.066007 15833 raft_consensus.cc:1261] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.067247 16946 consensus_queue.cc:1035] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.068637 16946 consensus_queue.cc:1035] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.072280 15832 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.072362 15570 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.072693 16984 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.073030 16984 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 110.685 ms
time per row: 0.0553423 ms
I20240502 14:00:32.138379 15427 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49520:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:32.143919 15551 tablet_service.cc:1467] Processing CreateTablet for tablet 186a946321e445b58b0097c658847ebe (DEFAULT_TABLE table=rebalance_test_table_8 [id=a148b6d170714c99a6024d0a3efa85c8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.144011 15811 tablet_service.cc:1467] Processing CreateTablet for tablet 186a946321e445b58b0097c658847ebe (DEFAULT_TABLE table=rebalance_test_table_8 [id=a148b6d170714c99a6024d0a3efa85c8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.144243 15551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 186a946321e445b58b0097c658847ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.144312 15811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 186a946321e445b58b0097c658847ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.145056 15550 tablet_service.cc:1467] Processing CreateTablet for tablet e51db70f5f284c4da3c1669fddb41b00 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a148b6d170714c99a6024d0a3efa85c8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.145344 15550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e51db70f5f284c4da3c1669fddb41b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.146358 17044 tablet_bootstrap.cc:492] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:32.146469 16831 tablet_bootstrap.cc:492] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:32.146993 15680 tablet_service.cc:1467] Processing CreateTablet for tablet 186a946321e445b58b0097c658847ebe (DEFAULT_TABLE table=rebalance_test_table_8 [id=a148b6d170714c99a6024d0a3efa85c8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.147266 15680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 186a946321e445b58b0097c658847ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.147375 16831 tablet_bootstrap.cc:654] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.147564 17044 tablet_bootstrap.cc:654] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.148418 15944 tablet_service.cc:1467] Processing CreateTablet for tablet e51db70f5f284c4da3c1669fddb41b00 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a148b6d170714c99a6024d0a3efa85c8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.148478 16831 tablet_bootstrap.cc:492] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:32.148557 16831 ts_tablet_manager.cc:1367] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:00:32.148687 15944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e51db70f5f284c4da3c1669fddb41b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.148983 16831 raft_consensus.cc:348] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:32.149111 16831 raft_consensus.cc:374] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.149147 16831 raft_consensus.cc:729] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.149269 16831 consensus_queue.cc:260] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:32.149551 16831 ts_tablet_manager.cc:1398] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:32.149551 15682 tablet_service.cc:1467] Processing CreateTablet for tablet e51db70f5f284c4da3c1669fddb41b00 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a148b6d170714c99a6024d0a3efa85c8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.149780 15682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e51db70f5f284c4da3c1669fddb41b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.149914 15943 tablet_service.cc:1467] Processing CreateTablet for tablet 7c57cab68b764db0917ed98e74757ff0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a148b6d170714c99a6024d0a3efa85c8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.150143 15943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c57cab68b764db0917ed98e74757ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.150648 16834 tablet_bootstrap.cc:492] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:32.151365 16834 tablet_bootstrap.cc:654] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.151896 15681 tablet_service.cc:1467] Processing CreateTablet for tablet 7c57cab68b764db0917ed98e74757ff0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a148b6d170714c99a6024d0a3efa85c8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.152127 15681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c57cab68b764db0917ed98e74757ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.153767 16829 tablet_bootstrap.cc:492] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:32.153842 16834 tablet_bootstrap.cc:492] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:32.153929 16834 ts_tablet_manager.cc:1367] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:00:32.154347 16834 raft_consensus.cc:348] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:32.154469 16834 raft_consensus.cc:374] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.154518 16834 raft_consensus.cc:729] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.154583 15811 tablet_service.cc:1467] Processing CreateTablet for tablet 7c57cab68b764db0917ed98e74757ff0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=a148b6d170714c99a6024d0a3efa85c8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.154660 16834 consensus_queue.cc:260] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:32.154811 15811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c57cab68b764db0917ed98e74757ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.154981 16834 ts_tablet_manager.cc:1398] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.155082 16829 tablet_bootstrap.cc:654] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.155148 16834 tablet_bootstrap.cc:492] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:32.156018 16834 tablet_bootstrap.cc:654] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.156883 16829 tablet_bootstrap.cc:492] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:32.156957 16829 ts_tablet_manager.cc:1367] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:32.157119 16831 tablet_bootstrap.cc:492] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:32.157364 16829 raft_consensus.cc:348] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:32.157475 16829 raft_consensus.cc:374] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.157513 16829 raft_consensus.cc:729] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.157634 16829 consensus_queue.cc:260] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:32.157919 16829 ts_tablet_manager.cc:1398] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.157980 16831 tablet_bootstrap.cc:654] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.158093 16829 tablet_bootstrap.cc:492] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:32.158993 16829 tablet_bootstrap.cc:654] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.159139 17044 tablet_bootstrap.cc:492] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:32.159245 17044 ts_tablet_manager.cc:1367] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20240502 14:00:32.159703 17044 raft_consensus.cc:348] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:32.159817 17044 raft_consensus.cc:374] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.159852 17044 raft_consensus.cc:729] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.159994 17044 consensus_queue.cc:260] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:32.160302 17044 ts_tablet_manager.cc:1398] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.160303 16834 tablet_bootstrap.cc:492] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:32.160377 16834 ts_tablet_manager.cc:1367] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:32.160413 16829 tablet_bootstrap.cc:492] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:32.160466 17044 tablet_bootstrap.cc:492] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:32.160481 16829 ts_tablet_manager.cc:1367] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:32.160786 16834 raft_consensus.cc:348] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.160877 16834 raft_consensus.cc:374] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.160907 16834 raft_consensus.cc:729] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.160885 16829 raft_consensus.cc:348] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:32.160976 16829 raft_consensus.cc:374] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.161006 16829 raft_consensus.cc:729] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.161022 16834 consensus_queue.cc:260] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.161113 16829 consensus_queue.cc:260] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:32.161355 17044 tablet_bootstrap.cc:654] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.161386 16829 ts_tablet_manager.cc:1398] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.161549 16829 tablet_bootstrap.cc:492] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:32.162464 16834 ts_tablet_manager.cc:1398] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:32.162479 16829 tablet_bootstrap.cc:654] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.163621 16831 tablet_bootstrap.cc:492] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:32.163729 16831 ts_tablet_manager.cc:1367] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240502 14:00:32.164141 17044 tablet_bootstrap.cc:492] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:32.164240 17044 ts_tablet_manager.cc:1367] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:32.164193 16831 raft_consensus.cc:348] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.164311 16831 raft_consensus.cc:374] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.164355 16831 raft_consensus.cc:729] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.164467 16831 consensus_queue.cc:260] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.164667 17044 raft_consensus.cc:348] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:32.164762 16831 ts_tablet_manager.cc:1398] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:32.164770 17044 raft_consensus.cc:374] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.164819 17044 raft_consensus.cc:729] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.164940 17044 consensus_queue.cc:260] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:32.165192 17044 ts_tablet_manager.cc:1398] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:32.165699 16829 tablet_bootstrap.cc:492] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:32.165782 16829 ts_tablet_manager.cc:1367] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:32.166184 16829 raft_consensus.cc:348] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.166262 16829 raft_consensus.cc:374] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.166339 16829 raft_consensus.cc:729] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.166463 16829 consensus_queue.cc:260] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.166744 16829 ts_tablet_manager.cc:1398] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.175745 16828 raft_consensus.cc:482] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:32.175851 16828 raft_consensus.cc:504] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:32.176246 16828 leader_election.cc:288] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719), 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:32.176529 15963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e51db70f5f284c4da3c1669fddb41b00" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" is_pre_election: true
I20240502 14:00:32.176661 15963 raft_consensus.cc:2454] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 0.
I20240502 14:00:32.176720 15570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e51db70f5f284c4da3c1669fddb41b00" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" is_pre_election: true
I20240502 14:00:32.176851 15570 raft_consensus.cc:2454] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 0.
I20240502 14:00:32.176941 15635 leader_election.cc:302] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:32.177143 16828 raft_consensus.cc:2781] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:32.177202 16828 raft_consensus.cc:482] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:32.177248 16828 raft_consensus.cc:3037] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.178107 16828 raft_consensus.cc:504] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:32.178467 16828 leader_election.cc:288] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 election: Requested vote from peers 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719), 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:32.178717 15963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e51db70f5f284c4da3c1669fddb41b00" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
I20240502 14:00:32.178802 15570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e51db70f5f284c4da3c1669fddb41b00" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
I20240502 14:00:32.178861 15963 raft_consensus.cc:3037] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.178910 15570 raft_consensus.cc:3037] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.179795 15963 raft_consensus.cc:2454] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 1.
I20240502 14:00:32.179798 15570 raft_consensus.cc:2454] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 1.
I20240502 14:00:32.180076 15637 leader_election.cc:302] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:32.180261 16828 raft_consensus.cc:2781] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:32.180344 16828 raft_consensus.cc:686] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Becoming Leader. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Running, Role: LEADER
I20240502 14:00:32.180477 16828 consensus_queue.cc:237] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } }
I20240502 14:00:32.182148 15427 catalog_manager.cc:5484] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8484181c4ece483c8b408817fab1a917 (127.8.86.130). New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } }
I20240502 14:00:32.251634 16753 raft_consensus.cc:482] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:32.251816 16753 raft_consensus.cc:504] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:32.252348 16753 leader_election.cc:288] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:32.252856 15832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "186a946321e445b58b0097c658847ebe" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf" is_pre_election: true
I20240502 14:00:32.252868 15700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "186a946321e445b58b0097c658847ebe" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917" is_pre_election: true
I20240502 14:00:32.252998 15832 raft_consensus.cc:2454] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 0.
I20240502 14:00:32.253026 15700 raft_consensus.cc:2454] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 0.
I20240502 14:00:32.253365 15507 leader_election.cc:302] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:32.253558 16753 raft_consensus.cc:2781] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:32.253623 16753 raft_consensus.cc:482] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:32.253664 16753 raft_consensus.cc:3037] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.254580 16753 raft_consensus.cc:504] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:32.254930 16753 leader_election.cc:288] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 election: Requested vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:32.255277 15700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "186a946321e445b58b0097c658847ebe" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917"
I20240502 14:00:32.255321 15832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "186a946321e445b58b0097c658847ebe" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf"
I20240502 14:00:32.255400 15700 raft_consensus.cc:3037] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.255421 15832 raft_consensus.cc:3037] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.256440 15832 raft_consensus.cc:2454] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 1.
I20240502 14:00:32.256457 15700 raft_consensus.cc:2454] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 1.
I20240502 14:00:32.256721 15507 leader_election.cc:302] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:32.256896 16753 raft_consensus.cc:2781] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:32.256983 16753 raft_consensus.cc:686] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Becoming Leader. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Running, Role: LEADER
I20240502 14:00:32.257134 16753 consensus_queue.cc:237] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:32.259158 15427 catalog_manager.cc:5484] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129). New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:32.435554 17043 raft_consensus.cc:482] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:32.435688 17043 raft_consensus.cc:504] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.436082 17043 leader_election.cc:288] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:32.436419 15700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c57cab68b764db0917ed98e74757ff0" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917" is_pre_election: true
I20240502 14:00:32.436533 15700 raft_consensus.cc:2454] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 0.
I20240502 14:00:32.436483 15963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c57cab68b764db0917ed98e74757ff0" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" is_pre_election: true
I20240502 14:00:32.436638 15963 raft_consensus.cc:2454] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 0.
I20240502 14:00:32.436836 15766 leader_election.cc:302] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82d7b7201cfc431d90baca1ad0a397cf, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:32.437052 17043 raft_consensus.cc:2781] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:32.437121 17043 raft_consensus.cc:482] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:32.437168 17043 raft_consensus.cc:3037] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.438040 17043 raft_consensus.cc:504] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.438373 17043 leader_election.cc:288] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 election: Requested vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:32.438658 15700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c57cab68b764db0917ed98e74757ff0" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917"
I20240502 14:00:32.438707 15963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c57cab68b764db0917ed98e74757ff0" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
I20240502 14:00:32.438786 15700 raft_consensus.cc:3037] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.438830 15963 raft_consensus.cc:3037] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.439548 15700 raft_consensus.cc:2454] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 1.
I20240502 14:00:32.439558 15963 raft_consensus.cc:2454] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 1.
I20240502 14:00:32.439885 15766 leader_election.cc:302] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82d7b7201cfc431d90baca1ad0a397cf, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:32.440136 17043 raft_consensus.cc:2781] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:32.440218 17043 raft_consensus.cc:686] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Becoming Leader. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Running, Role: LEADER
I20240502 14:00:32.440354 17043 consensus_queue.cc:237] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.442193 15427 catalog_manager.cc:5484] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131). New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } }
W20240502 14:00:32.561777 17057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:32.562053 17057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:32.565766 17057 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 14:00:32.599862 16828 consensus_queue.cc:1035] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.603771 16983 consensus_queue.cc:1035] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.653086 15700 raft_consensus.cc:1261] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.653187 15832 raft_consensus.cc:1261] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.653575 16753 consensus_queue.cc:1035] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.654278 16753 consensus_queue.cc:1035] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.676229 15963 raft_consensus.cc:1261] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.676597 16899 consensus_queue.cc:1035] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:32.698763 15700 raft_consensus.cc:1261] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:32.699193 16899 consensus_queue.cc:1035] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 115.227 ms
time per row: 0.0576135 ms
I20240502 14:00:32.725267 15425 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49520:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:32.731276 15811 tablet_service.cc:1467] Processing CreateTablet for tablet 1c004727e7c64f33bc9492184b77fde9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=17d53f89ea7b4f918ed8bb35d44f6401]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.731634 15811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c004727e7c64f33bc9492184b77fde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.731771 15551 tablet_service.cc:1467] Processing CreateTablet for tablet e48dda3e62e14e46b03d79648e3e2773 (DEFAULT_TABLE table=rebalance_test_table_9 [id=17d53f89ea7b4f918ed8bb35d44f6401]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.732041 15551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48dda3e62e14e46b03d79648e3e2773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.732331 15681 tablet_service.cc:1467] Processing CreateTablet for tablet 9870bc797f184a6284ff6ca4f78b3826 (DEFAULT_TABLE table=rebalance_test_table_9 [id=17d53f89ea7b4f918ed8bb35d44f6401]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.732420 15943 tablet_service.cc:1467] Processing CreateTablet for tablet 1c004727e7c64f33bc9492184b77fde9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=17d53f89ea7b4f918ed8bb35d44f6401]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.732604 15681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9870bc797f184a6284ff6ca4f78b3826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.732667 15943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c004727e7c64f33bc9492184b77fde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.734167 15550 tablet_service.cc:1467] Processing CreateTablet for tablet 1c004727e7c64f33bc9492184b77fde9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=17d53f89ea7b4f918ed8bb35d44f6401]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.734421 15550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c004727e7c64f33bc9492184b77fde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.734766 17113 tablet_bootstrap.cc:492] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:32.734902 15944 tablet_service.cc:1467] Processing CreateTablet for tablet e48dda3e62e14e46b03d79648e3e2773 (DEFAULT_TABLE table=rebalance_test_table_9 [id=17d53f89ea7b4f918ed8bb35d44f6401]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.735128 15944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48dda3e62e14e46b03d79648e3e2773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.735277 17112 tablet_bootstrap.cc:492] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:32.735721 17113 tablet_bootstrap.cc:654] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.736135 17112 tablet_bootstrap.cc:654] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.736418 17114 tablet_bootstrap.cc:492] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:32.736510 15812 tablet_service.cc:1467] Processing CreateTablet for tablet e48dda3e62e14e46b03d79648e3e2773 (DEFAULT_TABLE table=rebalance_test_table_9 [id=17d53f89ea7b4f918ed8bb35d44f6401]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.736718 15812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48dda3e62e14e46b03d79648e3e2773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.737166 17112 tablet_bootstrap.cc:492] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:32.737246 17112 ts_tablet_manager.cc:1367] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:32.737324 17114 tablet_bootstrap.cc:654] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.737818 17112 raft_consensus.cc:348] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.737949 17112 raft_consensus.cc:374] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.737915 15549 tablet_service.cc:1467] Processing CreateTablet for tablet 9870bc797f184a6284ff6ca4f78b3826 (DEFAULT_TABLE table=rebalance_test_table_9 [id=17d53f89ea7b4f918ed8bb35d44f6401]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.737987 17112 raft_consensus.cc:729] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.738152 15549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9870bc797f184a6284ff6ca4f78b3826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.738113 17112 consensus_queue.cc:260] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.738456 17112 ts_tablet_manager.cc:1398] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.738469 17111 tablet_bootstrap.cc:492] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:32.738631 17112 tablet_bootstrap.cc:492] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:32.739457 17112 tablet_bootstrap.cc:654] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.739617 17111 tablet_bootstrap.cc:654] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.741039 17112 tablet_bootstrap.cc:492] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:32.741114 17112 ts_tablet_manager.cc:1367] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:32.741307 15811 tablet_service.cc:1467] Processing CreateTablet for tablet 9870bc797f184a6284ff6ca4f78b3826 (DEFAULT_TABLE table=rebalance_test_table_9 [id=17d53f89ea7b4f918ed8bb35d44f6401]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.741534 15811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9870bc797f184a6284ff6ca4f78b3826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.741537 17112 raft_consensus.cc:348] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.741654 17112 raft_consensus.cc:374] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.741693 17112 raft_consensus.cc:729] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.741819 17112 consensus_queue.cc:260] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.742182 17111 tablet_bootstrap.cc:492] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:32.742269 17111 ts_tablet_manager.cc:1367] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:32.742777 17112 ts_tablet_manager.cc:1398] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:32.743484 17111 raft_consensus.cc:348] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.743592 17111 raft_consensus.cc:374] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.743626 17111 raft_consensus.cc:729] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.743767 17111 consensus_queue.cc:260] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.743870 17113 tablet_bootstrap.cc:492] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:32.743974 17113 ts_tablet_manager.cc:1367] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:00:32.744036 17111 ts_tablet_manager.cc:1398] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:00:32.744204 17111 tablet_bootstrap.cc:492] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:32.744395 17113 raft_consensus.cc:348] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.744503 17113 raft_consensus.cc:374] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.744541 17113 raft_consensus.cc:729] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.744657 17113 consensus_queue.cc:260] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.744776 17112 tablet_bootstrap.cc:492] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:00:32.744925 17113 ts_tablet_manager.cc:1398] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.745108 17113 tablet_bootstrap.cc:492] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:32.745697 17112 tablet_bootstrap.cc:654] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.746026 17113 tablet_bootstrap.cc:654] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.746784 17114 tablet_bootstrap.cc:492] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917: No bootstrap required, opened a new log
I20240502 14:00:32.746873 17114 ts_tablet_manager.cc:1367] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:00:32.747532 17114 raft_consensus.cc:348] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:32.747639 17114 raft_consensus.cc:374] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.747678 17114 raft_consensus.cc:729] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.747785 17111 tablet_bootstrap.cc:654] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.747790 17114 consensus_queue.cc:260] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:32.748072 17114 ts_tablet_manager.cc:1398] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:32.748332 17112 tablet_bootstrap.cc:492] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf: No bootstrap required, opened a new log
I20240502 14:00:32.748410 17112 ts_tablet_manager.cc:1367] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:32.748772 17111 tablet_bootstrap.cc:492] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:32.748865 17111 ts_tablet_manager.cc:1367] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:32.748821 17112 raft_consensus.cc:348] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:32.748919 17112 raft_consensus.cc:374] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.748958 17112 raft_consensus.cc:729] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.749083 17112 consensus_queue.cc:260] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:32.749289 17111 raft_consensus.cc:348] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.749351 17112 ts_tablet_manager.cc:1398] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.749389 17111 raft_consensus.cc:374] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.749429 17111 raft_consensus.cc:729] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.749541 17111 consensus_queue.cc:260] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.750027 17111 ts_tablet_manager.cc:1398] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.750267 17111 tablet_bootstrap.cc:492] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:00:32.750811 17113 tablet_bootstrap.cc:492] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd: No bootstrap required, opened a new log
I20240502 14:00:32.750895 17113 ts_tablet_manager.cc:1367] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:32.751278 17111 tablet_bootstrap.cc:654] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.752365 17113 raft_consensus.cc:348] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.752518 17113 raft_consensus.cc:374] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.752597 17113 raft_consensus.cc:729] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.752784 17113 consensus_queue.cc:260] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.753233 17113 ts_tablet_manager.cc:1398] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:32.754258 17111 tablet_bootstrap.cc:492] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf: No bootstrap required, opened a new log
I20240502 14:00:32.754359 17111 ts_tablet_manager.cc:1367] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:00:32.754709 17111 raft_consensus.cc:348] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:32.754789 17111 raft_consensus.cc:374] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.754823 17111 raft_consensus.cc:729] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.754948 17111 consensus_queue.cc:260] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:32.755592 17111 ts_tablet_manager.cc:1398] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:32.773087 16753 raft_consensus.cc:482] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:32.773190 16753 raft_consensus.cc:504] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:32.773536 16753 leader_election.cc:288] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:32.773844 15700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9870bc797f184a6284ff6ca4f78b3826" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917" is_pre_election: true
I20240502 14:00:32.773882 15832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9870bc797f184a6284ff6ca4f78b3826" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf" is_pre_election: true
I20240502 14:00:32.773958 15700 raft_consensus.cc:2454] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 0.
I20240502 14:00:32.774019 15832 raft_consensus.cc:2454] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 0.
I20240502 14:00:32.774269 15504 leader_election.cc:302] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 8484181c4ece483c8b408817fab1a917; no voters:
I20240502 14:00:32.774447 16753 raft_consensus.cc:2781] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:32.774502 16753 raft_consensus.cc:482] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:32.774542 16753 raft_consensus.cc:3037] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.775341 16753 raft_consensus.cc:504] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:32.775697 16753 leader_election.cc:288] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 election: Requested vote from peers 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:32.776010 15700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9870bc797f184a6284ff6ca4f78b3826" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8484181c4ece483c8b408817fab1a917"
I20240502 14:00:32.776028 15832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9870bc797f184a6284ff6ca4f78b3826" candidate_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf"
I20240502 14:00:32.776108 15700 raft_consensus.cc:3037] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.776113 15832 raft_consensus.cc:3037] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.776823 15832 raft_consensus.cc:2454] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 1.
I20240502 14:00:32.776830 15700 raft_consensus.cc:2454] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eb0e3a6c782469fa8110f6bb6a585cf in term 1.
I20240502 14:00:32.777078 15507 leader_election.cc:302] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:32.777276 16899 raft_consensus.cc:482] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:32.777361 16899 raft_consensus.cc:504] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.777740 16899 leader_election.cc:288] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237), 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:32.777864 16753 raft_consensus.cc:2781] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:32.777954 16753 raft_consensus.cc:686] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Becoming Leader. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Running, Role: LEADER
I20240502 14:00:32.778082 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c004727e7c64f33bc9492184b77fde9" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" is_pre_election: true
I20240502 14:00:32.778086 16753 consensus_queue.cc:237] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } }
I20240502 14:00:32.778086 15570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c004727e7c64f33bc9492184b77fde9" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" is_pre_election: true
I20240502 14:00:32.778230 15964 raft_consensus.cc:2454] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 0.
I20240502 14:00:32.778230 15570 raft_consensus.cc:2454] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 0.
I20240502 14:00:32.778564 15766 leader_election.cc:302] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:32.778726 16899 raft_consensus.cc:2781] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:32.778786 16899 raft_consensus.cc:482] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:32.778827 16899 raft_consensus.cc:3037] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.779589 16899 raft_consensus.cc:504] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.779886 16899 leader_election.cc:288] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 election: Requested vote from peers 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237), 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:32.780130 15422 catalog_manager.cc:5484] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129). New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:32.780210 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c004727e7c64f33bc9492184b77fde9" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
I20240502 14:00:32.780328 15964 raft_consensus.cc:3037] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.781173 15964 raft_consensus.cc:2454] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 1.
I20240502 14:00:32.781455 15570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c004727e7c64f33bc9492184b77fde9" candidate_uuid: "82d7b7201cfc431d90baca1ad0a397cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
I20240502 14:00:32.781561 15570 raft_consensus.cc:3037] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.782246 15570 raft_consensus.cc:2454] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82d7b7201cfc431d90baca1ad0a397cf in term 1.
I20240502 14:00:32.782532 15768 leader_election.cc:302] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:32.782724 16899 raft_consensus.cc:2781] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:32.782795 16899 raft_consensus.cc:686] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Becoming Leader. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Running, Role: LEADER
I20240502 14:00:32.782909 16899 consensus_queue.cc:237] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.784600 15422 catalog_manager.cc:5484] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131). New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:32.876607 16877 raft_consensus.cc:482] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:32.876736 16877 raft_consensus.cc:504] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.877120 16877 leader_election.cc:288] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531), 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:32.877439 15832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48dda3e62e14e46b03d79648e3e2773" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf" is_pre_election: true
I20240502 14:00:32.877506 15570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48dda3e62e14e46b03d79648e3e2773" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" is_pre_election: true
I20240502 14:00:32.877589 15832 raft_consensus.cc:2454] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 0.
I20240502 14:00:32.877648 15570 raft_consensus.cc:2454] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 0.
I20240502 14:00:32.877858 15900 leader_election.cc:302] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:32.878029 16877 raft_consensus.cc:2781] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:32.878091 16877 raft_consensus.cc:482] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:32.878127 16877 raft_consensus.cc:3037] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.882373 16877 raft_consensus.cc:504] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.882694 16877 leader_election.cc:288] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 election: Requested vote from peers 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531), 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:32.882992 15832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48dda3e62e14e46b03d79648e3e2773" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82d7b7201cfc431d90baca1ad0a397cf"
I20240502 14:00:32.883023 15570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48dda3e62e14e46b03d79648e3e2773" candidate_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
I20240502 14:00:32.883100 15832 raft_consensus.cc:3037] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.883123 15570 raft_consensus.cc:3037] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.883845 15832 raft_consensus.cc:2454] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 1.
I20240502 14:00:32.883855 15570 raft_consensus.cc:2454] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d0f8f311e834d81bd54b9f8e7923bcd in term 1.
I20240502 14:00:32.884135 15900 leader_election.cc:302] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 82d7b7201cfc431d90baca1ad0a397cf; no voters:
I20240502 14:00:32.884279 16877 raft_consensus.cc:2781] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:32.884366 16877 raft_consensus.cc:686] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Becoming Leader. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Running, Role: LEADER
I20240502 14:00:32.884500 16877 consensus_queue.cc:237] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } }
I20240502 14:00:32.886206 15422 catalog_manager.cc:5484] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132). New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } }
W20240502 14:00:33.002316 17127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:33.002593 17127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:33.005741 17127 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 14:00:33.087623 15700 raft_consensus.cc:1261] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:33.087716 15963 raft_consensus.cc:1261] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:33.087965 15833 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:33.088052 15832 raft_consensus.cc:1261] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:33.088133 17043 consensus_queue.cc:1035] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:33.088418 16882 consensus_queue.cc:1035] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:33.088585 15570 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:33.088547 16877 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:33.088843 15571 raft_consensus.cc:1261] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:33.089361 16882 consensus_queue.cc:1035] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:33.090533 16899 consensus_queue.cc:1035] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:33.091646 16877 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:33.118443 17089 mvcc.cc:205] Tried to move back new op lower bound from 7023240941922041856 to 7023240940660801536. Current Snapshot: MvccSnapshot[applied={T|T < 7023240941922041856 or (T in {7023240941922041856})}]
INSERT report
rows total: 2000
time total: 118.571 ms
time per row: 0.0592855 ms
I20240502 14:00:33.166005 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.133:42151
--local_ip_for_outbound_sockets=127.8.86.133
--tserver_master_addrs=127.8.86.190:35811
--webserver_port=38045
--webserver_interface=127.8.86.133
--builtin_ntp_servers=127.8.86.148:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:00:33.278820 17169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:33.279063 17169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:33.279110 17169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:33.282202 17169 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:00:33.282274 17169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:33.282418 17169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.133
I20240502 14:00:33.286413 17169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.133:42151
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.8.86.133
--webserver_port=38045
--tserver_master_addrs=127.8.86.190:35811
--never_fsync=true
--heap_profile_path=/tmp/kudu.17169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.133
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:33.287457 17169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:33.288671 17169 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:33.297305 17177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:33.297839 17174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:33.297997 17175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:33.298671 17169 server_base.cc:1034] running on GCE node
I20240502 14:00:33.299075 17169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:33.299633 17169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:33.300781 17169 hybrid_clock.cc:648] HybridClock initialized: now 1714658433300768 us; error 26 us; skew 500 ppm
I20240502 14:00:33.302781 17169 webserver.cc:460] Webserver started at http://127.8.86.133:38045/ using document root <none> and password file <none>
I20240502 14:00:33.303294 17169 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:33.303361 17169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:33.306712 17169 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:00:33.308508 17183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:33.309486 17169 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:33.309600 17169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f688370715454d6e814a78fcf85ac5c3"
format_stamp: "Formatted at 2024-05-02 14:00:23 on dist-test-slave-h6v0"
I20240502 14:00:33.310009 17169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:33.326956 17169 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:33.327508 17169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:33.327656 17169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:33.328130 17169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:33.328954 17169 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:33.328999 17169 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:33.329037 17169 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:33.329057 17169 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:33.337252 17169 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.133:42151
I20240502 14:00:33.337275 17296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.133:42151 every 8 connection(s)
I20240502 14:00:33.338235 17169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:00:33.341688 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 17169
I20240502 14:00:33.342096 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.134:34379
--local_ip_for_outbound_sockets=127.8.86.134
--tserver_master_addrs=127.8.86.190:35811
--webserver_port=33023
--webserver_interface=127.8.86.134
--builtin_ntp_servers=127.8.86.148:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:00:33.347225 17297 heartbeater.cc:347] Connected to a master server at 127.8.86.190:35811
I20240502 14:00:33.347447 17297 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:33.347985 17297 heartbeater.cc:511] Master 127.8.86.190:35811 requested a full tablet report, sending...
I20240502 14:00:33.349114 15426 ts_manager.cc:194] Re-registered known tserver with Master: f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
I20240502 14:00:33.349750 15426 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.133:59973
W20240502 14:00:33.451678 17300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:33.451897 17300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:33.451977 17300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:33.455253 17300 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:00:33.455333 17300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:33.455439 17300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.134
I20240502 14:00:33.459095 17300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.134:34379
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.8.86.134
--webserver_port=33023
--tserver_master_addrs=127.8.86.190:35811
--never_fsync=true
--heap_profile_path=/tmp/kudu.17300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.134
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:33.460004 17300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:33.461076 17300 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:33.469676 17306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:33.470089 17305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:33.470369 17308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:33.470993 17300 server_base.cc:1034] running on GCE node
I20240502 14:00:33.471365 17300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:33.471921 17300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:33.473071 17300 hybrid_clock.cc:648] HybridClock initialized: now 1714658433473051 us; error 40 us; skew 500 ppm
I20240502 14:00:33.475080 17300 webserver.cc:460] Webserver started at http://127.8.86.134:33023/ using document root <none> and password file <none>
I20240502 14:00:33.475634 17300 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:33.475723 17300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:33.479239 17300 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:00:33.481170 17314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:33.482246 17300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:33.482381 17300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "3c3658c946994e2ba9f41b64aa327d25"
format_stamp: "Formatted at 2024-05-02 14:00:23 on dist-test-slave-h6v0"
I20240502 14:00:33.482811 17300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:33.503069 17300 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:33.503763 17300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:33.503945 17300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:33.504492 17300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:33.505390 17300 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:33.505439 17300 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:33.505512 17300 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:33.505537 17300 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:33.514061 17300 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.134:34379
I20240502 14:00:33.514086 17427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.134:34379 every 8 connection(s)
I20240502 14:00:33.515172 17300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:00:33.518152 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 17300
I20240502 14:00:33.518699 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.135:46245
--local_ip_for_outbound_sockets=127.8.86.135
--tserver_master_addrs=127.8.86.190:35811
--webserver_port=35089
--webserver_interface=127.8.86.135
--builtin_ntp_servers=127.8.86.148:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:00:33.525189 17428 heartbeater.cc:347] Connected to a master server at 127.8.86.190:35811
I20240502 14:00:33.525470 17428 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:33.526223 17428 heartbeater.cc:511] Master 127.8.86.190:35811 requested a full tablet report, sending...
I20240502 14:00:33.527380 15426 ts_manager.cc:194] Re-registered known tserver with Master: 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379)
I20240502 14:00:33.527959 15426 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.134:48229
W20240502 14:00:33.638757 17431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:33.638984 17431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:33.639050 17431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:33.642330 17431 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:00:33.642410 17431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:33.642500 17431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.135
I20240502 14:00:33.646304 17431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.135:46245
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.8.86.135
--webserver_port=35089
--tserver_master_addrs=127.8.86.190:35811
--never_fsync=true
--heap_profile_path=/tmp/kudu.17431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.135
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:33.647233 17431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:33.648329 17431 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:33.656800 17437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:33.657260 17440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:33.657467 17438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:33.658139 17431 server_base.cc:1034] running on GCE node
I20240502 14:00:33.658612 17431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:33.659380 17431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:33.661491 17431 hybrid_clock.cc:648] HybridClock initialized: now 1714658433661473 us; error 32 us; skew 500 ppm
I20240502 14:00:33.663831 17431 webserver.cc:460] Webserver started at http://127.8.86.135:35089/ using document root <none> and password file <none>
I20240502 14:00:33.664398 17431 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:33.664472 17431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:33.667752 17431 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:00:33.669616 17446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:33.670822 17431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:33.670928 17431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "8fea283177394bd28d7eacf4fb744d59"
format_stamp: "Formatted at 2024-05-02 14:00:23 on dist-test-slave-h6v0"
I20240502 14:00:33.671316 17431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:33.693061 17431 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:33.693744 17431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:33.693923 17431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:33.694626 17431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:33.695519 17431 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:33.695564 17431 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:33.695606 17431 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:33.695644 17431 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:33.704867 17431 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.135:46245
I20240502 14:00:33.704903 17560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.135:46245 every 8 connection(s)
I20240502 14:00:33.705941 17431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:00:33.714857 17561 heartbeater.cc:347] Connected to a master server at 127.8.86.190:35811
I20240502 14:00:33.715075 17561 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:33.715595 17561 heartbeater.cc:511] Master 127.8.86.190:35811 requested a full tablet report, sending...
I20240502 14:00:33.716080 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 17431
I20240502 14:00:33.716562 8538 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4LMC7s/build/debug/bin/kudu
/tmp/dist-test-task4LMC7s/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.86.136:38631
--local_ip_for_outbound_sockets=127.8.86.136
--tserver_master_addrs=127.8.86.190:35811
--webserver_port=40225
--webserver_interface=127.8.86.136
--builtin_ntp_servers=127.8.86.148:37405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:00:33.716825 15426 ts_manager.cc:194] Re-registered known tserver with Master: 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245)
I20240502 14:00:33.717429 15426 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.135:41057
W20240502 14:00:33.826700 17564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:33.826970 17564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:33.827060 17564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:33.830231 17564 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:00:33.830330 17564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:33.830452 17564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.86.136
I20240502 14:00:33.834228 17564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.86.148:37405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.86.136:38631
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.8.86.136
--webserver_port=40225
--tserver_master_addrs=127.8.86.190:35811
--never_fsync=true
--heap_profile_path=/tmp/kudu.17564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.86.136
--log_dir=/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:33.835165 17564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:33.836244 17564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:33.844245 17571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:33.844650 17570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:33.845067 17573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:33.845731 17564 server_base.cc:1034] running on GCE node
I20240502 14:00:33.846083 17564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:33.846648 17564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:33.847803 17564 hybrid_clock.cc:648] HybridClock initialized: now 1714658433847786 us; error 33 us; skew 500 ppm
I20240502 14:00:33.849666 17564 webserver.cc:460] Webserver started at http://127.8.86.136:40225/ using document root <none> and password file <none>
I20240502 14:00:33.850205 17564 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:33.850303 17564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:33.853791 17564 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:33.855744 17579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:33.856750 17564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:33.856853 17564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "2254b3ee3db647598d781e467584f78d"
format_stamp: "Formatted at 2024-05-02 14:00:24 on dist-test-slave-h6v0"
I20240502 14:00:33.857262 17564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:33.877737 17564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:33.878342 17564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:33.878485 17564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:33.878966 17564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:33.879875 17564 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:33.879984 17564 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:33.880039 17564 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:33.880064 17564 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:33.888381 17564 rpc_server.cc:307] RPC server started. Bound to: 127.8.86.136:38631
I20240502 14:00:33.888388 17692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.86.136:38631 every 8 connection(s)
I20240502 14:00:33.889386 17564 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:00:33.892676 8538 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4LMC7s/build/debug/bin/kudu as pid 17564
I20240502 14:00:33.899784 17693 heartbeater.cc:347] Connected to a master server at 127.8.86.190:35811
I20240502 14:00:33.900040 17693 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:33.900698 17693 heartbeater.cc:511] Master 127.8.86.190:35811 requested a full tablet report, sending...
I20240502 14:00:33.901933 15426 ts_manager.cc:194] Re-registered known tserver with Master: 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631)
I20240502 14:00:33.902781 15426 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.86.136:51969
I20240502 14:00:34.297648 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:34.307211 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:34.321766 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.321825 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.333015 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.344182 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.349625 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.351817 17297 heartbeater.cc:503] Master 127.8.86.190:35811 was elected leader, sending a full tablet report...
I20240502 14:00:34.352841 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.353847 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.362221 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.373337 17484 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.373339 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.382200 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.387996 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.398553 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.403386 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.411145 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.429572 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.438602 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.455420 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:34.457218 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.458114 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.458752 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.473088 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.511178 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.520063 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:34.521703 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.530216 17428 heartbeater.cc:503] Master 127.8.86.190:35811 was elected leader, sending a full tablet report...
I20240502 14:00:34.533030 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.535198 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.553395 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.567384 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.575546 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.591924 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.592849 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.594244 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:34.595191 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.601923 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.644662 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.645560 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.665629 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.719559 17561 heartbeater.cc:503] Master 127.8.86.190:35811 was elected leader, sending a full tablet report...
I20240502 14:00:34.830214 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:34.832187 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.843353 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.846537 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.848245 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.849213 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.855085 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.876776 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.878257 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.883683 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.895833 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:34.904466 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.906136 17693 heartbeater.cc:503] Master 127.8.86.190:35811 was elected leader, sending a full tablet report...
I20240502 14:00:34.920701 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.921087 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:34.927686 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.933279 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.936009 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.941543 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.949314 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.952658 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.954301 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:34.959450 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:34.984091 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.038833 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.059574 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.068925 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.078817 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.081794 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.096000 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.098619 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:35.123858 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.142935 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.173808 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.177599 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:35.192276 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.193434 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.199319 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.219882 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.220911 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.235971 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.328656 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:35.333273 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.342446 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.347487 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.349329 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.353689 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.364382 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.373939 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.391211 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.394433 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:35.396502 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.429478 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.429666 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.445166 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.451318 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.455796 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:35.464697 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.465751 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.467114 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.468147 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.469887 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.490088 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.503613 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.512135 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.576809 15570 consensus_queue.cc:237] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:35.582273 15832 raft_consensus.cc:1261] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.582952 15963 raft_consensus.cc:1261] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.585281 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:35.591611 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.592941 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 14:00:35.594024 15504 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 619a8ec66ba74ecea689c17b4734253a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.595024 18402 consensus_queue.cc:1035] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.595757 18402 consensus_queue.cc:1035] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.598242 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.604400 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.621089 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.621589 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.627602 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.640810 17167 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:35.641479 15833 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:35.651865 15426 catalog_manager.cc:5484] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:35.653287 15963 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:35.672088 15570 consensus_queue.cc:237] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:35.674041 15963 raft_consensus.cc:1261] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.674566 17167 consensus_queue.cc:1035] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.679153 15700 raft_consensus.cc:1261] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.679592 17167 consensus_queue.cc:1035] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:35.681403 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f562ad00ec34a8d9184671ed55d84d1. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.688771 17167 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:35.691984 15702 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:35.695844 15428 catalog_manager.cc:5484] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:35.698993 15570 consensus_queue.cc:237] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:35.699532 15963 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:35.700824 15833 raft_consensus.cc:1261] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.702113 18400 consensus_queue.cc:1035] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:35.702613 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 186a946321e445b58b0097c658847ebe. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.703079 15700 raft_consensus.cc:1261] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.703220 15570 consensus_queue.cc:237] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:35.703439 18400 consensus_queue.cc:1035] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.704989 15832 raft_consensus.cc:1261] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:35.705439 18400 consensus_queue.cc:1035] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:00:35.705857 15504 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 619a8ec66ba74ecea689c17b4734253a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.707222 18401 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:35.707787 15832 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:35.710498 15963 raft_consensus.cc:1261] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:00:35.710837 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 619a8ec66ba74ecea689c17b4734253a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.711022 18401 consensus_queue.cc:1035] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:35.711941 15963 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:35.719046 18401 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:35.721948 15832 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:35.730737 15700 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:35.736601 15428 catalog_manager.cc:5484] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 4 to 5, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:35.741786 15832 consensus_queue.cc:237] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:35.742019 15422 catalog_manager.cc:5484] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:35.743954 15702 raft_consensus.cc:1261] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.743954 15963 raft_consensus.cc:1261] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.744370 16899 consensus_queue.cc:1035] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.745088 16899 consensus_queue.cc:1035] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.750103 15832 consensus_queue.cc:237] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:35.752374 15964 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.752677 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.752756 15702 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.753202 16899 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.753582 16899 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.754907 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.757916 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:35.760476 15701 consensus_queue.cc:237] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:35.761314 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.762090 15832 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.762197 15570 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.762578 18174 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.762912 18174 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:35.778126 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03dd735113ab4ea6bdb696b11a4efd7f. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:35.783298 15766 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22eef55211ca4f65979022d2f4ae829f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.783833 18234 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:35.784833 15963 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:35.786993 15700 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:35.787525 15425 catalog_manager.cc:5484] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:35.788815 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.789810 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:35.794414 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.796403 18506 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:35.798072 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.799499 15702 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:35.801451 15422 catalog_manager.cc:5484] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:35.828241 18447 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:35.829435 15964 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:35.832543 15570 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:35.836175 15428 catalog_manager.cc:5484] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:35.839905 15833 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:35.844866 15571 consensus_queue.cc:237] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
W20240502 14:00:35.846711 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76f07464d9bd49a9b1b9d58f6a180cf8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.846874 15963 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:35.847262 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfdcf1124e8a474facc14c45e8963568. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.847348 18402 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.847584 15702 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.848301 18402 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.849972 15833 consensus_queue.cc:237] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
W20240502 14:00:35.851958 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02d3da05d82340178e27218c2ce56b67. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.852259 15571 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.852691 18492 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.852881 15700 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.853281 18492 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.854619 18402 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:35.856279 18492 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:35.856981 15700 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:35.857417 15963 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:35.860469 15428 catalog_manager.cc:5484] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:35.860782 15425 catalog_manager.cc:5484] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:35.863101 15571 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:35.872601 15702 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:35.882789 15963 consensus_queue.cc:237] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:35.884636 15571 consensus_queue.cc:237] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:35.885401 15570 consensus_queue.cc:237] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:35.886353 15832 raft_consensus.cc:1261] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.886579 15569 raft_consensus.cc:1261] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.886372 15833 raft_consensus.cc:1261] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:35.886781 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d81f14b7c1d1468196d0fe6d67fe93a7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.886915 15702 raft_consensus.cc:1261] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.887142 17876 consensus_queue.cc:1035] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.887665 18401 consensus_queue.cc:1035] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.887981 18401 consensus_queue.cc:1035] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.888442 15833 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:35.888752 15504 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 436c127af87c49e8b7bf9b6571f7dc30. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.889060 15702 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.889511 18400 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.889717 17876 consensus_queue.cc:1035] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.890350 18400 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:35.890615 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a5e33f337bd47b2a6b11b29a5eacad8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.892751 18400 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:35.894019 17167 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:35.894564 15833 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:35.896689 15831 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:35.896849 15700 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:35.897001 15428 catalog_manager.cc:5484] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:35.898427 15702 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:35.902180 15426 catalog_manager.cc:5484] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:35.904489 17876 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:35.907585 15428 catalog_manager.cc:5484] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:35.908022 15832 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:35.910084 15569 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:35.913786 15963 consensus_queue.cc:237] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:35.917537 15569 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.918722 17876 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.920243 15832 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.920634 17876 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.924047 17876 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:35.926263 15569 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:35.928563 15702 consensus_queue.cc:237] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:35.928915 15425 catalog_manager.cc:5484] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:35.930559 15832 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.930919 15963 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.931378 18174 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.933867 15832 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:35.934108 18174 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.940539 15702 consensus_queue.cc:237] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
W20240502 14:00:35.941363 15897 consensus_peers.cc:485] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 387bdd8ae3b34f588e2170b061a5bc05. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.942181 15569 raft_consensus.cc:1261] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.942598 18447 consensus_queue.cc:1035] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.944360 18502 raft_consensus.cc:2932] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:35.944828 15569 raft_consensus.cc:2932] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
W20240502 14:00:35.946638 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48fbc670ec504d26a11bf1b9fbd78981. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:35.946746 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b07f4db6f82e4a56accdbf1bbbc5f965. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.947582 15425 catalog_manager.cc:5484] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:35.947530 15964 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:35.949873 15963 raft_consensus.cc:1261] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.950318 18502 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:35.951615 18174 consensus_queue.cc:1035] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.953034 15963 raft_consensus.cc:2932] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:35.956578 15426 catalog_manager.cc:5484] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:35.964409 15832 consensus_queue.cc:237] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:35.966136 15569 raft_consensus.cc:1261] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.966163 15963 raft_consensus.cc:1261] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.966567 16899 consensus_queue.cc:1035] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.967382 16899 consensus_queue.cc:1035] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.971240 18234 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:35.972460 15832 consensus_queue.cc:237] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:35.972851 15569 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:35.974313 15702 raft_consensus.cc:1261] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.974763 15964 raft_consensus.cc:1261] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:35.975309 18507 consensus_queue.cc:1035] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:35.975322 15425 catalog_manager.cc:5484] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:00:35.976310 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c57cab68b764db0917ed98e74757ff0. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.976322 18507 consensus_queue.cc:1035] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:35.978435 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c004727e7c64f33bc9492184b77fde9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:35.980119 15963 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:35.984103 16899 raft_consensus.cc:2932] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:35.985859 15832 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:35.986719 15964 raft_consensus.cc:2932] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:35.988940 15420 catalog_manager.cc:5484] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:35.993775 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:35.996742 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:35.997653 15702 raft_consensus.cc:2932] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.004248 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:36.012589 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:36.019636 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:36.035225 15569 consensus_queue.cc:237] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:36.038007 15963 raft_consensus.cc:1261] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.039873 18402 consensus_queue.cc:1035] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:00:36.040371 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f562ad00ec34a8d9184671ed55d84d1. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.041157 15702 raft_consensus.cc:1261] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:00:36.042732 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f562ad00ec34a8d9184671ed55d84d1. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.042877 18401 consensus_queue.cc:1035] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.044394 18402 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:36.045712 15963 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:36.049137 15426 catalog_manager.cc:5484] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:36.049535 15702 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:36.052917 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:36.055754 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:36.060443 15569 consensus_queue.cc:237] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:36.062270 15702 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.062400 15963 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:00:36.062402 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfdcf1124e8a474facc14c45e8963568. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.062695 18401 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.062819 17167 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.063503 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:00:36.064891 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfdcf1124e8a474facc14c45e8963568. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.064764 17167 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:36.065421 15964 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:36.066159 15702 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:36.068989 15420 catalog_manager.cc:5484] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:36.103689 15702 consensus_queue.cc:237] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:36.106094 15832 raft_consensus.cc:1261] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:36.107162 15569 raft_consensus.cc:1261] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:36.107643 18447 consensus_queue.cc:1035] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:36.110073 18447 consensus_queue.cc:1035] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:36.113765 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f80c8590e994a10a9b7c6c174e28173. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.121603 15964 consensus_queue.cc:237] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:36.123350 15832 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:36.123287 18447 raft_consensus.cc:2932] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.123790 18584 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:36.125041 15570 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:36.125512 18584 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:36.125700 15833 raft_consensus.cc:2932] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.126092 15569 raft_consensus.cc:2932] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.128998 15420 catalog_manager.cc:5484] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
W20240502 14:00:36.129253 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e48dda3e62e14e46b03d79648e3e2773. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.131213 18700 ts_tablet_manager.cc:902] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:36.134222 18566 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:36.134809 15570 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:36.132263 18700 tablet_copy_client.cc:308] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:36.137916 15428 catalog_manager.cc:5484] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:36.151012 15832 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:36.156419 15591 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet 619a8ec66ba74ecea689c17b4734253a from peer 8fea283177394bd28d7eacf4fb744d59 ({username='slave'} at 127.8.86.135:41945)
I20240502 14:00:36.156610 15591 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet 619a8ec66ba74ecea689c17b4734253a from peer 8fea283177394bd28d7eacf4fb744d59 at {username='slave'} at 127.8.86.135:41945: session id = 8fea283177394bd28d7eacf4fb744d59-619a8ec66ba74ecea689c17b4734253a
I20240502 14:00:36.158831 15591 tablet_copy_source_session.cc:205] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.159816 15570 consensus_queue.cc:237] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:36.161934 15702 raft_consensus.cc:1261] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:36.162444 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9870bc797f184a6284ff6ca4f78b3826. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.162565 18401 consensus_queue.cc:1035] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:36.162828 15832 raft_consensus.cc:1261] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:36.163941 18401 consensus_queue.cc:1035] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:36.165068 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:36.165050 18401 raft_consensus.cc:2932] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.167378 18700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619a8ec66ba74ecea689c17b4734253a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.167639 15702 raft_consensus.cc:2932] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.170805 15425 catalog_manager.cc:5484] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.181666 15833 consensus_queue.cc:237] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:36.182683 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 14:00:36.183705 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2886dfbec1e141198b0cf7a435fac498. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.184383 15570 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:36.184850 18506 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:36.186733 15964 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:36.187158 18506 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:36.188697 15962 consensus_queue.cc:237] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:36.188856 15833 consensus_queue.cc:237] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:36.190440 15702 raft_consensus.cc:1261] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:36.190759 15767 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22eef55211ca4f65979022d2f4ae829f. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:36.190842 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d92e5885bf94c3e8e0e4dee40017699. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.191015 15702 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.190997 18566 consensus_queue.cc:1035] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:36.191474 15766 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22eef55211ca4f65979022d2f4ae829f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.191577 18506 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.192694 15962 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.193102 15833 raft_consensus.cc:1261] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:36.193238 18492 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.193464 18566 consensus_queue.cc:1035] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:36.193548 18492 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:36.193955 18584 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.195327 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:36.196455 15702 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.197032 15420 catalog_manager.cc:5484] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:36.198601 15425 catalog_manager.cc:5484] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:36.207597 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:36.207926 15962 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:36.208994 15702 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:36.210956 18506 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:36.213449 15570 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:36.213604 15425 catalog_manager.cc:5484] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:36.220597 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:36.224402 18763 ts_tablet_manager.cc:902] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d: Initiating tablet copy from peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:36.226269 15831 consensus_queue.cc:237] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:36.227958 15702 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:36.228335 18763 tablet_copy_client.cc:308] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:42261
W20240502 14:00:36.228744 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22eef55211ca4f65979022d2f4ae829f. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:36.229039 15767 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22eef55211ca4f65979022d2f4ae829f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.230379 18492 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:00:36.230640 15766 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22eef55211ca4f65979022d2f4ae829f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.231985 15962 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:36.232326 18492 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:36.232837 18492 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.235220 15722 tablet_copy_service.cc:140] P 8484181c4ece483c8b408817fab1a917: Received BeginTabletCopySession request for tablet 76f07464d9bd49a9b1b9d58f6a180cf8 from peer 2254b3ee3db647598d781e467584f78d ({username='slave'} at 127.8.86.136:42777)
I20240502 14:00:36.235365 15722 tablet_copy_service.cc:161] P 8484181c4ece483c8b408817fab1a917: Beginning new tablet copy session on tablet 76f07464d9bd49a9b1b9d58f6a180cf8 from peer 2254b3ee3db647598d781e467584f78d at {username='slave'} at 127.8.86.136:42777: session id = 2254b3ee3db647598d781e467584f78d-76f07464d9bd49a9b1b9d58f6a180cf8
I20240502 14:00:36.235667 15426 catalog_manager.cc:5484] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 5 to 6, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:36.237378 15722 tablet_copy_source_session.cc:205] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.237607 15702 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.241904 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:36.243052 15962 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.248829 18763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f07464d9bd49a9b1b9d58f6a180cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.249667 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:36.257702 18700 tablet_copy_client.cc:783] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.257987 18700 tablet_copy_client.cc:647] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.259735 15964 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:36.260174 18700 tablet_copy_client.cc:523] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.261595 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:36.266127 15833 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.269770 18700 tablet_bootstrap.cc:492] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59: Bootstrap starting.
I20240502 14:00:36.275547 18763 tablet_copy_client.cc:783] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.275935 18763 tablet_copy_client.cc:647] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.282073 18700 log.cc:824] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:36.284006 15570 consensus_queue.cc:237] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:36.284164 18751 ts_tablet_manager.cc:902] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:36.285022 18751 tablet_copy_client.cc:308] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:36.286548 15702 raft_consensus.cc:1261] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:36.289592 15832 raft_consensus.cc:1261] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:36.289908 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb9f100600a246708457280d91826fce. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.290215 17167 consensus_queue.cc:1035] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:36.291199 18400 consensus_queue.cc:1035] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:36.292287 18773 ts_tablet_manager.cc:902] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:36.292245 15832 raft_consensus.cc:2932] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.292857 18773 tablet_copy_client.cc:308] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:36.295329 18763 tablet_copy_client.cc:523] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.297565 18763 tablet_bootstrap.cc:492] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d: Bootstrap starting.
I20240502 14:00:36.298480 18788 ts_tablet_manager.cc:902] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:36.299033 18788 tablet_copy_client.cc:308] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
I20240502 14:00:36.303087 15591 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet 186a946321e445b58b0097c658847ebe from peer 3c3658c946994e2ba9f41b64aa327d25 ({username='slave'} at 127.8.86.134:43305)
I20240502 14:00:36.303134 15590 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet 619a8ec66ba74ecea689c17b4734253a from peer 3c3658c946994e2ba9f41b64aa327d25 ({username='slave'} at 127.8.86.134:43305)
I20240502 14:00:36.303201 15591 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet 186a946321e445b58b0097c658847ebe from peer 3c3658c946994e2ba9f41b64aa327d25 at {username='slave'} at 127.8.86.134:43305: session id = 3c3658c946994e2ba9f41b64aa327d25-186a946321e445b58b0097c658847ebe
I20240502 14:00:36.303313 15590 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet 619a8ec66ba74ecea689c17b4734253a from peer 3c3658c946994e2ba9f41b64aa327d25 at {username='slave'} at 127.8.86.134:43305: session id = 3c3658c946994e2ba9f41b64aa327d25-619a8ec66ba74ecea689c17b4734253a
I20240502 14:00:36.304131 15591 tablet_copy_source_session.cc:205] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.304338 15590 tablet_copy_source_session.cc:205] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.305097 18763 log.cc:824] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:36.307704 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 03dd735113ab4ea6bdb696b11a4efd7f from peer 8fea283177394bd28d7eacf4fb744d59 ({username='slave'} at 127.8.86.135:39871)
I20240502 14:00:36.307839 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 03dd735113ab4ea6bdb696b11a4efd7f from peer 8fea283177394bd28d7eacf4fb744d59 at {username='slave'} at 127.8.86.135:39871: session id = 8fea283177394bd28d7eacf4fb744d59-03dd735113ab4ea6bdb696b11a4efd7f
I20240502 14:00:36.309233 15853 tablet_copy_source_session.cc:205] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.313293 18751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 186a946321e445b58b0097c658847ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.314607 18773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619a8ec66ba74ecea689c17b4734253a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.319182 18402 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:36.319787 15832 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:36.322934 15425 catalog_manager.cc:5484] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:36.323369 18751 tablet_copy_client.cc:783] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.323597 18751 tablet_copy_client.cc:647] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.323719 15702 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:36.324651 18773 tablet_copy_client.cc:783] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.324826 18773 tablet_copy_client.cc:647] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.331414 18751 tablet_copy_client.cc:523] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.333801 18751 tablet_bootstrap.cc:492] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap starting.
I20240502 14:00:36.336196 18821 ts_tablet_manager.cc:902] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59: Initiating tablet copy from peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:36.336869 18821 tablet_copy_client.cc:308] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.132:45719
I20240502 14:00:36.338627 18788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03dd735113ab4ea6bdb696b11a4efd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.340957 18788 tablet_copy_client.cc:783] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.341156 18788 tablet_copy_client.cc:647] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.341275 18773 tablet_copy_client.cc:523] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.342566 18773 tablet_bootstrap.cc:492] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap starting.
I20240502 14:00:36.342895 15984 tablet_copy_service.cc:140] P 4d0f8f311e834d81bd54b9f8e7923bcd: Received BeginTabletCopySession request for tablet 1a5e33f337bd47b2a6b11b29a5eacad8 from peer 8fea283177394bd28d7eacf4fb744d59 ({username='slave'} at 127.8.86.135:44181)
I20240502 14:00:36.343032 15984 tablet_copy_service.cc:161] P 4d0f8f311e834d81bd54b9f8e7923bcd: Beginning new tablet copy session on tablet 1a5e33f337bd47b2a6b11b29a5eacad8 from peer 8fea283177394bd28d7eacf4fb744d59 at {username='slave'} at 127.8.86.135:44181: session id = 8fea283177394bd28d7eacf4fb744d59-1a5e33f337bd47b2a6b11b29a5eacad8
I20240502 14:00:36.344372 15984 tablet_copy_source_session.cc:205] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.345964 18821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5e33f337bd47b2a6b11b29a5eacad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.348115 18821 tablet_copy_client.cc:783] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.348243 18821 tablet_copy_client.cc:647] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.350787 15570 consensus_queue.cc:237] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:36.352950 18788 tablet_copy_client.cc:523] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.353821 15962 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:36.354647 18788 tablet_bootstrap.cc:492] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59: Bootstrap starting.
I20240502 14:00:36.355449 15702 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:00:36.356441 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfdcf1124e8a474facc14c45e8963568. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:36.356689 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfdcf1124e8a474facc14c45e8963568. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:36.357195 15506 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfdcf1124e8a474facc14c45e8963568. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.357923 18402 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:36.358253 18402 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:36.360484 18653 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.361068 15964 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.364477 15832 consensus_queue.cc:237] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:36.364611 15428 catalog_manager.cc:5484] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 5 to 6, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
W20240502 14:00:36.366644 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2886dfbec1e141198b0cf7a435fac498. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.366631 15700 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.367331 15570 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.367461 15964 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.367744 16899 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.368475 18507 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:00:36.370234 15766 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2886dfbec1e141198b0cf7a435fac498. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.371485 18751 log.cc:824] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:36.372468 16899 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:36.376039 15570 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:36.379474 15425 catalog_manager.cc:5484] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:36.381911 18821 tablet_copy_client.cc:523] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.383627 18821 tablet_bootstrap.cc:492] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap starting.
I20240502 14:00:36.388778 15964 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:36.400211 18832 ts_tablet_manager.cc:902] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3: Initiating tablet copy from peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:36.401046 18832 tablet_copy_client.cc:308] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:42261
I20240502 14:00:36.401517 18831 ts_tablet_manager.cc:902] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:36.402786 18831 tablet_copy_client.cc:308] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
I20240502 14:00:36.404713 18751 tablet_bootstrap.cc:492] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.405114 18751 tablet_bootstrap.cc:492] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap complete.
I20240502 14:00:36.405493 18751 ts_tablet_manager.cc:1367] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25: Time spent bootstrapping tablet: real 0.072s user 0.031s sys 0.005s
I20240502 14:00:36.406119 15570 consensus_queue.cc:237] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:36.408236 18751 raft_consensus.cc:348] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
W20240502 14:00:36.408488 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0cd21dc897b41aca9118be6df48cc0c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.408501 18751 raft_consensus.cc:729] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c3658c946994e2ba9f41b64aa327d25, State: Initialized, Role: LEARNER
I20240502 14:00:36.408577 15832 raft_consensus.cc:1261] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:36.408782 15964 raft_consensus.cc:1261] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:36.408587 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 02d3da05d82340178e27218c2ce56b67 from peer 2254b3ee3db647598d781e467584f78d ({username='slave'} at 127.8.86.136:53895)
I20240502 14:00:36.408898 18751 consensus_queue.cc:260] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:36.408970 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 02d3da05d82340178e27218c2ce56b67 from peer 2254b3ee3db647598d781e467584f78d at {username='slave'} at 127.8.86.136:53895: session id = 2254b3ee3db647598d781e467584f78d-02d3da05d82340178e27218c2ce56b67
I20240502 14:00:36.409183 18400 consensus_queue.cc:1035] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:36.409804 18751 ts_tablet_manager.cc:1398] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20240502 14:00:36.409900 15853 tablet_copy_source_session.cc:205] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.411022 18831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d3da05d82340178e27218c2ce56b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.411674 15590 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session 3c3658c946994e2ba9f41b64aa327d25-186a946321e445b58b0097c658847ebe received from {username='slave'} at 127.8.86.134:43305
I20240502 14:00:36.411762 15590 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session 3c3658c946994e2ba9f41b64aa327d25-186a946321e445b58b0097c658847ebe on tablet 186a946321e445b58b0097c658847ebe with peer 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:36.411772 18400 consensus_queue.cc:1035] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:36.413022 18751 ts_tablet_manager.cc:902] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:36.413486 18751 tablet_copy_client.cc:308] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:36.414312 15590 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet 436c127af87c49e8b7bf9b6571f7dc30 from peer 3c3658c946994e2ba9f41b64aa327d25 ({username='slave'} at 127.8.86.134:43305)
I20240502 14:00:36.414407 15590 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet 436c127af87c49e8b7bf9b6571f7dc30 from peer 3c3658c946994e2ba9f41b64aa327d25 at {username='slave'} at 127.8.86.134:43305: session id = 3c3658c946994e2ba9f41b64aa327d25-436c127af87c49e8b7bf9b6571f7dc30
I20240502 14:00:36.415333 15590 tablet_copy_source_session.cc:205] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.416257 18751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436c127af87c49e8b7bf9b6571f7dc30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.416183 17167 raft_consensus.cc:2932] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.418740 18751 tablet_copy_client.cc:783] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.418893 18751 tablet_copy_client.cc:647] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.419162 15964 raft_consensus.cc:2932] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.419584 15428 catalog_manager.cc:5484] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:36.420047 18751 tablet_copy_client.cc:523] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.421181 18751 tablet_bootstrap.cc:492] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap starting.
I20240502 14:00:36.421468 15832 raft_consensus.cc:2932] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.434463 18852 ts_tablet_manager.cc:902] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25: Initiating tablet copy from peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:36.435168 18852 tablet_copy_client.cc:308] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.132:45719
I20240502 14:00:36.437476 18831 tablet_copy_client.cc:783] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.437675 18831 tablet_copy_client.cc:647] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.442060 15984 tablet_copy_service.cc:140] P 4d0f8f311e834d81bd54b9f8e7923bcd: Received BeginTabletCopySession request for tablet 387bdd8ae3b34f588e2170b061a5bc05 from peer 3c3658c946994e2ba9f41b64aa327d25 ({username='slave'} at 127.8.86.134:37349)
I20240502 14:00:36.442164 15984 tablet_copy_service.cc:161] P 4d0f8f311e834d81bd54b9f8e7923bcd: Beginning new tablet copy session on tablet 387bdd8ae3b34f588e2170b061a5bc05 from peer 3c3658c946994e2ba9f41b64aa327d25 at {username='slave'} at 127.8.86.134:37349: session id = 3c3658c946994e2ba9f41b64aa327d25-387bdd8ae3b34f588e2170b061a5bc05
I20240502 14:00:36.443260 15984 tablet_copy_source_session.cc:205] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.447331 18751 tablet_bootstrap.cc:492] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.447657 18751 tablet_bootstrap.cc:492] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap complete.
I20240502 14:00:36.447854 18751 ts_tablet_manager.cc:1367] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25: Time spent bootstrapping tablet: real 0.027s user 0.021s sys 0.004s
I20240502 14:00:36.448496 18751 raft_consensus.cc:348] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:36.448633 18751 raft_consensus.cc:729] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c3658c946994e2ba9f41b64aa327d25, State: Initialized, Role: LEARNER
I20240502 14:00:36.448729 18751 consensus_queue.cc:260] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:36.449112 18751 ts_tablet_manager.cc:1398] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 14:00:36.449788 15590 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session 3c3658c946994e2ba9f41b64aa327d25-436c127af87c49e8b7bf9b6571f7dc30 received from {username='slave'} at 127.8.86.134:43305
I20240502 14:00:36.449865 15590 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session 3c3658c946994e2ba9f41b64aa327d25-436c127af87c49e8b7bf9b6571f7dc30 on tablet 436c127af87c49e8b7bf9b6571f7dc30 with peer 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:36.451504 15832 consensus_queue.cc:237] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:36.452387 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:36.453589 15700 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:00:36.453845 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02d3da05d82340178e27218c2ce56b67. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.454345 15569 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.454540 18506 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:00:36.454664 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.455056 18506 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.457046 18507 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:36.457893 18841 ts_tablet_manager.cc:902] T b07f4db6f82e4a56accdbf1bbbc5f965 P f688370715454d6e814a78fcf85ac5c3: Initiating tablet copy from peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:36.457975 15700 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:36.460325 15569 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:36.461092 15426 catalog_manager.cc:5484] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:36.464401 18851 ts_tablet_manager.cc:902] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:36.465201 18851 tablet_copy_client.cc:308] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
I20240502 14:00:36.465744 18871 ts_tablet_manager.cc:902] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:36.465924 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 7c57cab68b764db0917ed98e74757ff0 from peer 8fea283177394bd28d7eacf4fb744d59 ({username='slave'} at 127.8.86.135:39871)
I20240502 14:00:36.466032 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 7c57cab68b764db0917ed98e74757ff0 from peer 8fea283177394bd28d7eacf4fb744d59 at {username='slave'} at 127.8.86.135:39871: session id = 8fea283177394bd28d7eacf4fb744d59-7c57cab68b764db0917ed98e74757ff0
I20240502 14:00:36.466922 15853 tablet_copy_source_session.cc:205] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.472385 18831 tablet_copy_client.cc:523] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.474269 18831 tablet_bootstrap.cc:492] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d: Bootstrap starting.
I20240502 14:00:36.478448 18841 tablet_copy_client.cc:308] T b07f4db6f82e4a56accdbf1bbbc5f965 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:42261
I20240502 14:00:36.478883 18871 tablet_copy_client.cc:308] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
I20240502 14:00:36.479247 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:36.480239 18851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c57cab68b764db0917ed98e74757ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.483464 18852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387bdd8ae3b34f588e2170b061a5bc05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.483999 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:36.485298 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:36.485503 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:36.486043 18852 tablet_copy_client.cc:783] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.486256 18852 tablet_copy_client.cc:647] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.486929 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:36.492327 18897 ts_tablet_manager.cc:902] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:36.493046 18897 tablet_copy_client.cc:308] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:36.498903 15700 consensus_queue.cc:237] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:36.501009 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:36.501467 15569 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:36.503854 18502 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:36.506214 15964 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:36.506844 18502 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:36.509771 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 1c004727e7c64f33bc9492184b77fde9 from peer f688370715454d6e814a78fcf85ac5c3 ({username='slave'} at 127.8.86.133:37489)
I20240502 14:00:36.509905 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 1c004727e7c64f33bc9492184b77fde9 from peer f688370715454d6e814a78fcf85ac5c3 at {username='slave'} at 127.8.86.133:37489: session id = f688370715454d6e814a78fcf85ac5c3-1c004727e7c64f33bc9492184b77fde9
I20240502 14:00:36.510836 15853 tablet_copy_source_session.cc:205] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.511632 15590 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet 1f562ad00ec34a8d9184671ed55d84d1 from peer f688370715454d6e814a78fcf85ac5c3 ({username='slave'} at 127.8.86.133:50351)
I20240502 14:00:36.511790 15590 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet 1f562ad00ec34a8d9184671ed55d84d1 from peer f688370715454d6e814a78fcf85ac5c3 at {username='slave'} at 127.8.86.133:50351: session id = f688370715454d6e814a78fcf85ac5c3-1f562ad00ec34a8d9184671ed55d84d1
I20240502 14:00:36.512693 15590 tablet_copy_source_session.cc:205] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.516093 18897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f562ad00ec34a8d9184671ed55d84d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.520722 18871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c004727e7c64f33bc9492184b77fde9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:00:36.522702 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7981c3f46d8483caae82fbf2296e762. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.524330 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:36.531494 18897 tablet_copy_client.cc:783] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.531733 18897 tablet_copy_client.cc:647] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.536432 18897 tablet_copy_client.cc:523] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.538576 18897 tablet_bootstrap.cc:492] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3: Bootstrap starting.
I20240502 14:00:36.545059 18917 ts_tablet_manager.cc:902] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:36.545680 18917 tablet_copy_client.cc:308] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:36.550766 15832 consensus_queue.cc:237] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
W20240502 14:00:36.552549 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c004727e7c64f33bc9492184b77fde9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.553134 15590 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet d81f14b7c1d1468196d0fe6d67fe93a7 from peer 8fea283177394bd28d7eacf4fb744d59 ({username='slave'} at 127.8.86.135:41945)
I20240502 14:00:36.553257 15590 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet d81f14b7c1d1468196d0fe6d67fe93a7 from peer 8fea283177394bd28d7eacf4fb744d59 at {username='slave'} at 127.8.86.135:41945: session id = 8fea283177394bd28d7eacf4fb744d59-d81f14b7c1d1468196d0fe6d67fe93a7
I20240502 14:00:36.554189 15590 tablet_copy_source_session.cc:205] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.555212 18917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81f14b7c1d1468196d0fe6d67fe93a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.555281 15570 raft_consensus.cc:1261] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.555685 18896 consensus_queue.cc:1035] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.557394 18928 ts_tablet_manager.cc:902] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:36.557965 18928 tablet_copy_client.cc:308] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:36.558574 15962 raft_consensus.cc:1261] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.558852 18926 ts_tablet_manager.cc:902] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d: Initiating tablet copy from peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:36.559230 18174 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.559732 18926 tablet_copy_client.cc:308] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.132:45719
I20240502 14:00:36.562250 18896 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
W20240502 14:00:36.563953 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c004727e7c64f33bc9492184b77fde9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.564291 18492 consensus_queue.cc:1035] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.564635 15569 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.565411 15962 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:36.565436 15425 catalog_manager.cc:5484] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:36.566208 15571 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:36.566735 15428 catalog_manager.cc:5484] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:36.568862 18851 tablet_copy_client.cc:783] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.569042 15590 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet 1f562ad00ec34a8d9184671ed55d84d1 from peer 2254b3ee3db647598d781e467584f78d ({username='slave'} at 127.8.86.136:58333)
I20240502 14:00:36.569121 15590 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet 1f562ad00ec34a8d9184671ed55d84d1 from peer 2254b3ee3db647598d781e467584f78d at {username='slave'} at 127.8.86.136:58333: session id = 2254b3ee3db647598d781e467584f78d-1f562ad00ec34a8d9184671ed55d84d1
I20240502 14:00:36.569998 15590 tablet_copy_source_session.cc:205] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.570526 18851 tablet_copy_client.cc:647] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.574091 15984 tablet_copy_service.cc:140] P 4d0f8f311e834d81bd54b9f8e7923bcd: Received BeginTabletCopySession request for tablet e48dda3e62e14e46b03d79648e3e2773 from peer 2254b3ee3db647598d781e467584f78d ({username='slave'} at 127.8.86.136:55499)
I20240502 14:00:36.574187 15984 tablet_copy_service.cc:161] P 4d0f8f311e834d81bd54b9f8e7923bcd: Beginning new tablet copy session on tablet e48dda3e62e14e46b03d79648e3e2773 from peer 2254b3ee3db647598d781e467584f78d at {username='slave'} at 127.8.86.136:55499: session id = 2254b3ee3db647598d781e467584f78d-e48dda3e62e14e46b03d79648e3e2773
I20240502 14:00:36.575170 15984 tablet_copy_source_session.cc:205] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.575218 18917 tablet_copy_client.cc:783] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.575381 18917 tablet_copy_client.cc:647] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.575029 15964 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.576598 18926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48dda3e62e14e46b03d79648e3e2773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.576798 18917 tablet_copy_client.cc:523] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.578465 18917 tablet_bootstrap.cc:492] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap starting.
I20240502 14:00:36.579329 18926 tablet_copy_client.cc:783] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.579696 18926 tablet_copy_client.cc:647] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.580601 18928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f562ad00ec34a8d9184671ed55d84d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.583290 18852 tablet_copy_client.cc:523] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.584911 18852 tablet_bootstrap.cc:492] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap starting.
I20240502 14:00:36.586346 18926 tablet_copy_client.cc:523] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.586509 18928 tablet_copy_client.cc:783] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.586786 18851 tablet_copy_client.cc:523] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.587680 18928 tablet_copy_client.cc:647] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.589051 18851 tablet_bootstrap.cc:492] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap starting.
I20240502 14:00:36.587724 18926 tablet_bootstrap.cc:492] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d: Bootstrap starting.
I20240502 14:00:36.589681 18928 tablet_copy_client.cc:523] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.590981 18928 tablet_bootstrap.cc:492] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d: Bootstrap starting.
I20240502 14:00:36.593511 15722 tablet_copy_service.cc:140] P 8484181c4ece483c8b408817fab1a917: Received BeginTabletCopySession request for tablet 48fbc670ec504d26a11bf1b9fbd78981 from peer f688370715454d6e814a78fcf85ac5c3 ({username='slave'} at 127.8.86.133:54505)
I20240502 14:00:36.593626 15722 tablet_copy_service.cc:161] P 8484181c4ece483c8b408817fab1a917: Beginning new tablet copy session on tablet 48fbc670ec504d26a11bf1b9fbd78981 from peer f688370715454d6e814a78fcf85ac5c3 at {username='slave'} at 127.8.86.133:54505: session id = f688370715454d6e814a78fcf85ac5c3-48fbc670ec504d26a11bf1b9fbd78981
I20240502 14:00:36.594650 15722 tablet_copy_source_session.cc:205] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.594998 15721 tablet_copy_service.cc:140] P 8484181c4ece483c8b408817fab1a917: Received BeginTabletCopySession request for tablet b07f4db6f82e4a56accdbf1bbbc5f965 from peer f688370715454d6e814a78fcf85ac5c3 ({username='slave'} at 127.8.86.133:54505)
I20240502 14:00:36.595103 15721 tablet_copy_service.cc:161] P 8484181c4ece483c8b408817fab1a917: Beginning new tablet copy session on tablet b07f4db6f82e4a56accdbf1bbbc5f965 from peer f688370715454d6e814a78fcf85ac5c3 at {username='slave'} at 127.8.86.133:54505: session id = f688370715454d6e814a78fcf85ac5c3-b07f4db6f82e4a56accdbf1bbbc5f965
I20240502 14:00:36.595933 15721 tablet_copy_source_session.cc:205] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.596997 18832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48fbc670ec504d26a11bf1b9fbd78981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.599313 18832 tablet_copy_client.cc:783] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.599453 18832 tablet_copy_client.cc:647] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.600903 18832 tablet_copy_client.cc:523] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.601608 18821 tablet_bootstrap.cc:492] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.602324 18832 tablet_bootstrap.cc:492] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3: Bootstrap starting.
I20240502 14:00:36.602461 18871 tablet_copy_client.cc:783] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.602699 18871 tablet_copy_client.cc:647] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.606087 18841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b07f4db6f82e4a56accdbf1bbbc5f965. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.607270 18871 tablet_copy_client.cc:523] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.608258 18841 tablet_copy_client.cc:783] T b07f4db6f82e4a56accdbf1bbbc5f965 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.608389 18841 tablet_copy_client.cc:647] T b07f4db6f82e4a56accdbf1bbbc5f965 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.608738 18871 tablet_bootstrap.cc:492] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3: Bootstrap starting.
I20240502 14:00:36.609704 18841 tablet_copy_client.cc:523] T b07f4db6f82e4a56accdbf1bbbc5f965 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.611078 18841 tablet_bootstrap.cc:492] T b07f4db6f82e4a56accdbf1bbbc5f965 P f688370715454d6e814a78fcf85ac5c3: Bootstrap starting.
I20240502 14:00:36.613020 18821 tablet_bootstrap.cc:492] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap complete.
I20240502 14:00:36.613377 18821 ts_tablet_manager.cc:1367] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59: Time spent bootstrapping tablet: real 0.230s user 0.029s sys 0.001s
I20240502 14:00:36.620015 15964 consensus_queue.cc:237] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:36.621044 18858 ts_tablet_manager.cc:902] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25: Initiating tablet copy from peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:36.621536 18858 tablet_copy_client.cc:308] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:42261
I20240502 14:00:36.596962 18897 log.cc:824] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:36.623310 15570 raft_consensus.cc:1261] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:36.623195 18821 raft_consensus.cc:348] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:36.623459 18821 raft_consensus.cc:729] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8fea283177394bd28d7eacf4fb744d59, State: Initialized, Role: LEARNER
I20240502 14:00:36.623845 18821 consensus_queue.cc:260] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:36.624361 18566 consensus_queue.cc:1035] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:36.624835 18821 ts_tablet_manager.cc:1398] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240502 14:00:36.626762 15721 tablet_copy_service.cc:140] P 8484181c4ece483c8b408817fab1a917: Received BeginTabletCopySession request for tablet 1f80c8590e994a10a9b7c6c174e28173 from peer 3c3658c946994e2ba9f41b64aa327d25 ({username='slave'} at 127.8.86.134:44409)
I20240502 14:00:36.626868 15721 tablet_copy_service.cc:161] P 8484181c4ece483c8b408817fab1a917: Beginning new tablet copy session on tablet 1f80c8590e994a10a9b7c6c174e28173 from peer 3c3658c946994e2ba9f41b64aa327d25 at {username='slave'} at 127.8.86.134:44409: session id = 3c3658c946994e2ba9f41b64aa327d25-1f80c8590e994a10a9b7c6c174e28173
I20240502 14:00:36.626909 15700 raft_consensus.cc:1261] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:36.627772 15721 tablet_copy_source_session.cc:205] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.628718 18858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f80c8590e994a10a9b7c6c174e28173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.629472 18566 consensus_queue.cc:1035] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:36.632273 15898 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 837e8b1235a44131bdaef24b763d1cac. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.638408 18831 tablet_bootstrap.cc:492] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.638826 18831 tablet_bootstrap.cc:492] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d: Bootstrap complete.
I20240502 14:00:36.639235 18831 ts_tablet_manager.cc:1367] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d: Time spent bootstrapping tablet: real 0.165s user 0.023s sys 0.006s
I20240502 14:00:36.641794 18831 raft_consensus.cc:348] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:36.642238 18831 raft_consensus.cc:729] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2254b3ee3db647598d781e467584f78d, State: Initialized, Role: LEARNER
I20240502 14:00:36.642984 15832 consensus_queue.cc:237] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:36.644076 15984 tablet_copy_service.cc:342] P 4d0f8f311e834d81bd54b9f8e7923bcd: Request end of tablet copy session 8fea283177394bd28d7eacf4fb744d59-1a5e33f337bd47b2a6b11b29a5eacad8 received from {username='slave'} at 127.8.86.135:44181
I20240502 14:00:36.644184 15984 tablet_copy_service.cc:433] P 4d0f8f311e834d81bd54b9f8e7923bcd: ending tablet copy session 8fea283177394bd28d7eacf4fb744d59-1a5e33f337bd47b2a6b11b29a5eacad8 on tablet 1a5e33f337bd47b2a6b11b29a5eacad8 with peer 8fea283177394bd28d7eacf4fb744d59
I20240502 14:00:36.647320 15964 raft_consensus.cc:1261] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.647650 17382 raft_consensus.cc:1203] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 14:00:36.648046 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.648468 15702 raft_consensus.cc:1261] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.649042 18700 tablet_bootstrap.cc:492] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.649214 16899 consensus_queue.cc:1035] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.649514 16899 consensus_queue.cc:1035] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.649466 18700 tablet_bootstrap.cc:492] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59: Bootstrap complete.
I20240502 14:00:36.650126 18700 ts_tablet_manager.cc:1367] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59: Time spent bootstrapping tablet: real 0.381s user 0.032s sys 0.005s
I20240502 14:00:36.652567 18492 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.654345 18858 tablet_copy_client.cc:783] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.654515 18858 tablet_copy_client.cc:647] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.655990 15420 catalog_manager.cc:5484] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:36.656263 15702 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.658829 15964 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.659504 18831 consensus_queue.cc:260] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:36.658187 18977 ts_tablet_manager.cc:902] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:36.660732 18977 tablet_copy_client.cc:308] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
I20240502 14:00:36.661286 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 22eef55211ca4f65979022d2f4ae829f from peer 2254b3ee3db647598d781e467584f78d ({username='slave'} at 127.8.86.136:53895)
I20240502 14:00:36.661394 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 22eef55211ca4f65979022d2f4ae829f from peer 2254b3ee3db647598d781e467584f78d at {username='slave'} at 127.8.86.136:53895: session id = 2254b3ee3db647598d781e467584f78d-22eef55211ca4f65979022d2f4ae829f
I20240502 14:00:36.662314 15853 tablet_copy_source_session.cc:205] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.667644 18831 ts_tablet_manager.cc:1398] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d: Time spent starting tablet: real 0.028s user 0.004s sys 0.000s
I20240502 14:00:36.668617 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session 2254b3ee3db647598d781e467584f78d-02d3da05d82340178e27218c2ce56b67 received from {username='slave'} at 127.8.86.136:53895
I20240502 14:00:36.669510 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session 2254b3ee3db647598d781e467584f78d-02d3da05d82340178e27218c2ce56b67 on tablet 02d3da05d82340178e27218c2ce56b67 with peer 2254b3ee3db647598d781e467584f78d
I20240502 14:00:36.671190 15702 consensus_queue.cc:237] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:36.673431 15571 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:00:36.673534 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76f07464d9bd49a9b1b9d58f6a180cf8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.673643 15832 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:00:36.673938 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.674145 18502 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.674357 18605 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.675711 18566 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:36.675985 18977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22eef55211ca4f65979022d2f4ae829f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.678479 18977 tablet_copy_client.cc:783] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.678598 18858 tablet_copy_client.cc:523] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.678710 18977 tablet_copy_client.cc:647] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.679774 18751 ts_tablet_manager.cc:902] T 9870bc797f184a6284ff6ca4f78b3826 P 3c3658c946994e2ba9f41b64aa327d25: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:36.680119 18858 tablet_bootstrap.cc:492] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap starting.
I20240502 14:00:36.680321 18751 tablet_copy_client.cc:308] T 9870bc797f184a6284ff6ca4f78b3826 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:36.680508 15570 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:36.683848 15590 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet 9870bc797f184a6284ff6ca4f78b3826 from peer 3c3658c946994e2ba9f41b64aa327d25 ({username='slave'} at 127.8.86.134:43305)
I20240502 14:00:36.683954 15590 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet 9870bc797f184a6284ff6ca4f78b3826 from peer 3c3658c946994e2ba9f41b64aa327d25 at {username='slave'} at 127.8.86.134:43305: session id = 3c3658c946994e2ba9f41b64aa327d25-9870bc797f184a6284ff6ca4f78b3826
I20240502 14:00:36.684824 15590 tablet_copy_source_session.cc:205] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.686406 18751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9870bc797f184a6284ff6ca4f78b3826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.686427 18605 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.686581 15420 catalog_manager.cc:5484] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:36.688556 18751 tablet_copy_client.cc:783] T 9870bc797f184a6284ff6ca4f78b3826 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.688696 18751 tablet_copy_client.cc:647] T 9870bc797f184a6284ff6ca4f78b3826 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.688597 15570 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.691372 15700 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
W20240502 14:00:36.693687 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03dd735113ab4ea6bdb696b11a4efd7f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.695573 15426 catalog_manager.cc:5484] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.696151 15832 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.703802 18977 tablet_copy_client.cc:523] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.709749 18700 raft_consensus.cc:348] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:36.709939 18700 raft_consensus.cc:729] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8fea283177394bd28d7eacf4fb744d59, State: Initialized, Role: LEARNER
I20240502 14:00:36.710074 18700 consensus_queue.cc:260] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:36.710557 18700 ts_tablet_manager.cc:1398] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59: Time spent starting tablet: real 0.060s user 0.001s sys 0.000s
I20240502 14:00:36.711150 15590 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session 8fea283177394bd28d7eacf4fb744d59-619a8ec66ba74ecea689c17b4734253a received from {username='slave'} at 127.8.86.135:41945
I20240502 14:00:36.711236 15590 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session 8fea283177394bd28d7eacf4fb744d59-619a8ec66ba74ecea689c17b4734253a on tablet 619a8ec66ba74ecea689c17b4734253a with peer 8fea283177394bd28d7eacf4fb744d59
I20240502 14:00:36.717365 18928 tablet_bootstrap.cc:492] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.718547 18751 tablet_copy_client.cc:523] T 9870bc797f184a6284ff6ca4f78b3826 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.720062 18751 tablet_bootstrap.cc:492] T 9870bc797f184a6284ff6ca4f78b3826 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap starting.
I20240502 14:00:36.724117 18700 ts_tablet_manager.cc:902] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59: Initiating tablet copy from peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:36.724567 18700 tablet_copy_client.cc:308] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.132:45719
I20240502 14:00:36.725039 15984 tablet_copy_service.cc:140] P 4d0f8f311e834d81bd54b9f8e7923bcd: Received BeginTabletCopySession request for tablet 2d92e5885bf94c3e8e0e4dee40017699 from peer 8fea283177394bd28d7eacf4fb744d59 ({username='slave'} at 127.8.86.135:44181)
I20240502 14:00:36.725152 15984 tablet_copy_service.cc:161] P 4d0f8f311e834d81bd54b9f8e7923bcd: Beginning new tablet copy session on tablet 2d92e5885bf94c3e8e0e4dee40017699 from peer 8fea283177394bd28d7eacf4fb744d59 at {username='slave'} at 127.8.86.135:44181: session id = 8fea283177394bd28d7eacf4fb744d59-2d92e5885bf94c3e8e0e4dee40017699
I20240502 14:00:36.726150 15984 tablet_copy_source_session.cc:205] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.726401 18928 tablet_bootstrap.cc:492] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d: Bootstrap complete.
I20240502 14:00:36.726653 18928 ts_tablet_manager.cc:1367] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d: Time spent bootstrapping tablet: real 0.136s user 0.028s sys 0.002s
I20240502 14:00:36.727185 18700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d92e5885bf94c3e8e0e4dee40017699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.727370 18928 raft_consensus.cc:348] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:36.727500 18928 raft_consensus.cc:729] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2254b3ee3db647598d781e467584f78d, State: Initialized, Role: LEARNER
I20240502 14:00:36.727612 18928 consensus_queue.cc:260] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:36.729604 18700 tablet_copy_client.cc:783] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.729765 18700 tablet_copy_client.cc:647] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.730249 18832 tablet_bootstrap.cc:492] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.730661 18832 tablet_bootstrap.cc:492] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3: Bootstrap complete.
I20240502 14:00:36.731006 18832 ts_tablet_manager.cc:1367] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3: Time spent bootstrapping tablet: real 0.129s user 0.028s sys 0.002s
I20240502 14:00:36.733564 18928 ts_tablet_manager.cc:1398] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240502 14:00:36.734558 15590 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session 2254b3ee3db647598d781e467584f78d-1f562ad00ec34a8d9184671ed55d84d1 received from {username='slave'} at 127.8.86.136:58333
I20240502 14:00:36.734642 15590 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session 2254b3ee3db647598d781e467584f78d-1f562ad00ec34a8d9184671ed55d84d1 on tablet 1f562ad00ec34a8d9184671ed55d84d1 with peer 2254b3ee3db647598d781e467584f78d
I20240502 14:00:36.734735 18841 tablet_bootstrap.cc:492] T b07f4db6f82e4a56accdbf1bbbc5f965 P f688370715454d6e814a78fcf85ac5c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.734987 19001 ts_tablet_manager.cc:902] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:36.735108 18841 tablet_bootstrap.cc:492] T b07f4db6f82e4a56accdbf1bbbc5f965 P f688370715454d6e814a78fcf85ac5c3: Bootstrap complete.
I20240502 14:00:36.735881 19001 tablet_copy_client.cc:308] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
I20240502 14:00:36.736673 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 22eef55211ca4f65979022d2f4ae829f from peer f688370715454d6e814a78fcf85ac5c3 ({username='slave'} at 127.8.86.133:37489)
I20240502 14:00:36.736779 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 22eef55211ca4f65979022d2f4ae829f from peer f688370715454d6e814a78fcf85ac5c3 at {username='slave'} at 127.8.86.133:37489: session id = f688370715454d6e814a78fcf85ac5c3-22eef55211ca4f65979022d2f4ae829f
I20240502 14:00:36.736909 18977 tablet_bootstrap.cc:492] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d: Bootstrap starting.
I20240502 14:00:36.737027 18841 ts_tablet_manager.cc:1367] T b07f4db6f82e4a56accdbf1bbbc5f965 P f688370715454d6e814a78fcf85ac5c3: Time spent bootstrapping tablet: real 0.126s user 0.027s sys 0.003s
I20240502 14:00:36.737704 15853 tablet_copy_source_session.cc:205] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.737982 18841 raft_consensus.cc:348] T b07f4db6f82e4a56accdbf1bbbc5f965 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:36.738265 18841 raft_consensus.cc:729] T b07f4db6f82e4a56accdbf1bbbc5f965 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f688370715454d6e814a78fcf85ac5c3, State: Initialized, Role: LEARNER
I20240502 14:00:36.738821 18841 consensus_queue.cc:260] T b07f4db6f82e4a56accdbf1bbbc5f965 P f688370715454d6e814a78fcf85ac5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:36.739782 18841 ts_tablet_manager.cc:1398] T b07f4db6f82e4a56accdbf1bbbc5f965 P f688370715454d6e814a78fcf85ac5c3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:36.740553 15721 tablet_copy_service.cc:342] P 8484181c4ece483c8b408817fab1a917: Request end of tablet copy session f688370715454d6e814a78fcf85ac5c3-b07f4db6f82e4a56accdbf1bbbc5f965 received from {username='slave'} at 127.8.86.133:54505
I20240502 14:00:36.740662 15721 tablet_copy_service.cc:433] P 8484181c4ece483c8b408817fab1a917: ending tablet copy session f688370715454d6e814a78fcf85ac5c3-b07f4db6f82e4a56accdbf1bbbc5f965 on tablet b07f4db6f82e4a56accdbf1bbbc5f965 with peer f688370715454d6e814a78fcf85ac5c3
I20240502 14:00:36.738425 18832 raft_consensus.cc:348] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:36.741309 18832 raft_consensus.cc:729] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f688370715454d6e814a78fcf85ac5c3, State: Initialized, Role: LEARNER
I20240502 14:00:36.741416 18832 consensus_queue.cc:260] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:36.741781 18832 ts_tablet_manager.cc:1398] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20240502 14:00:36.742210 15721 tablet_copy_service.cc:342] P 8484181c4ece483c8b408817fab1a917: Request end of tablet copy session f688370715454d6e814a78fcf85ac5c3-48fbc670ec504d26a11bf1b9fbd78981 received from {username='slave'} at 127.8.86.133:54505
I20240502 14:00:36.742306 15721 tablet_copy_service.cc:433] P 8484181c4ece483c8b408817fab1a917: ending tablet copy session f688370715454d6e814a78fcf85ac5c3-48fbc670ec504d26a11bf1b9fbd78981 on tablet 48fbc670ec504d26a11bf1b9fbd78981 with peer f688370715454d6e814a78fcf85ac5c3
I20240502 14:00:36.742517 19001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22eef55211ca4f65979022d2f4ae829f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.766644 18700 tablet_copy_client.cc:523] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.768412 19001 tablet_copy_client.cc:783] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.768646 19001 tablet_copy_client.cc:647] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.771200 15570 consensus_queue.cc:237] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:36.774071 17382 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.774505 15700 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:00:36.775063 15506 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 436c127af87c49e8b7bf9b6571f7dc30. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.775612 18400 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.775897 18400 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.776844 15832 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.778499 18700 tablet_bootstrap.cc:492] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap starting.
I20240502 14:00:36.779809 18401 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.781476 18401 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.783422 15832 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.784400 18928 ts_tablet_manager.cc:902] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:36.784581 15700 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.784950 18928 tablet_copy_client.cc:308] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
I20240502 14:00:36.786087 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 2886dfbec1e141198b0cf7a435fac498 from peer 2254b3ee3db647598d781e467584f78d ({username='slave'} at 127.8.86.136:53895)
I20240502 14:00:36.786186 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 2886dfbec1e141198b0cf7a435fac498 from peer 2254b3ee3db647598d781e467584f78d at {username='slave'} at 127.8.86.136:53895: session id = 2254b3ee3db647598d781e467584f78d-2886dfbec1e141198b0cf7a435fac498
I20240502 14:00:36.786703 15421 catalog_manager.cc:5484] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 4 to 5, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.787073 15853 tablet_copy_source_session.cc:205] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.787812 18917 tablet_bootstrap.cc:492] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.788136 18917 tablet_bootstrap.cc:492] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap complete.
I20240502 14:00:36.788303 18917 ts_tablet_manager.cc:1367] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59: Time spent bootstrapping tablet: real 0.210s user 0.027s sys 0.000s
I20240502 14:00:36.788708 18928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2886dfbec1e141198b0cf7a435fac498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.788800 18917 raft_consensus.cc:348] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:36.788920 18917 raft_consensus.cc:729] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8fea283177394bd28d7eacf4fb744d59, State: Initialized, Role: LEARNER
I20240502 14:00:36.789017 18917 consensus_queue.cc:260] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:36.790108 18917 ts_tablet_manager.cc:1398] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:36.791303 18928 tablet_copy_client.cc:783] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.791457 18928 tablet_copy_client.cc:647] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.792102 15590 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session 8fea283177394bd28d7eacf4fb744d59-d81f14b7c1d1468196d0fe6d67fe93a7 received from {username='slave'} at 127.8.86.135:41945
I20240502 14:00:36.792186 15590 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session 8fea283177394bd28d7eacf4fb744d59-d81f14b7c1d1468196d0fe6d67fe93a7 on tablet d81f14b7c1d1468196d0fe6d67fe93a7 with peer 8fea283177394bd28d7eacf4fb744d59
I20240502 14:00:36.793648 18400 raft_consensus.cc:1050] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf: attempting to promote NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 to VOTER
I20240502 14:00:36.794153 18400 consensus_queue.cc:237] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:36.795854 15832 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:00:36.797139 15506 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 436c127af87c49e8b7bf9b6571f7dc30. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.799533 18402 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:36.800531 17381 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:36.801093 18402 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.802225 17381 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.802871 18928 tablet_copy_client.cc:523] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.803133 15700 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:36.803473 18401 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:36.804162 18928 tablet_bootstrap.cc:492] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d: Bootstrap starting.
I20240502 14:00:36.805481 18401 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.808372 15832 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.808677 15700 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.811533 17381 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:36.811278 15426 catalog_manager.cc:5484] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:36.813030 18926 tablet_bootstrap.cc:492] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.813311 18926 tablet_bootstrap.cc:492] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d: Bootstrap complete.
I20240502 14:00:36.813472 18926 ts_tablet_manager.cc:1367] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d: Time spent bootstrapping tablet: real 0.226s user 0.019s sys 0.007s
I20240502 14:00:36.813921 18926 raft_consensus.cc:348] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:36.814025 18926 raft_consensus.cc:729] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2254b3ee3db647598d781e467584f78d, State: Initialized, Role: LEARNER
I20240502 14:00:36.814116 18926 consensus_queue.cc:260] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:36.822456 15570 consensus_queue.cc:237] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }
I20240502 14:00:36.823871 15832 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:36.823905 15702 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:36.824267 17167 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:36.824930 17167 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:36.826017 18858 tablet_bootstrap.cc:492] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.826670 18653 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:36.827046 15702 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:36.829393 15415 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 436c127af87c49e8b7bf9b6571f7dc30 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:36.829639 15832 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:36.831840 15700 consensus_queue.cc:237] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:36.833029 15426 catalog_manager.cc:5484] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 6 to 7, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) evicted. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
W20240502 14:00:36.834614 15635 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48fbc670ec504d26a11bf1b9fbd78981. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.835145 18926 ts_tablet_manager.cc:1398] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20240502 14:00:36.831934 18897 tablet_bootstrap.cc:492] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.835567 18897 tablet_bootstrap.cc:492] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3: Bootstrap complete.
I20240502 14:00:36.835992 18858 tablet_bootstrap.cc:492] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap complete.
I20240502 14:00:36.836144 17250 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.836239 18897 ts_tablet_manager.cc:1367] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3: Time spent bootstrapping tablet: real 0.298s user 0.037s sys 0.000s
I20240502 14:00:36.836712 15984 tablet_copy_service.cc:342] P 4d0f8f311e834d81bd54b9f8e7923bcd: Request end of tablet copy session 2254b3ee3db647598d781e467584f78d-e48dda3e62e14e46b03d79648e3e2773 received from {username='slave'} at 127.8.86.136:55499
I20240502 14:00:36.836788 15984 tablet_copy_service.cc:433] P 4d0f8f311e834d81bd54b9f8e7923bcd: ending tablet copy session 2254b3ee3db647598d781e467584f78d-e48dda3e62e14e46b03d79648e3e2773 on tablet e48dda3e62e14e46b03d79648e3e2773 with peer 2254b3ee3db647598d781e467584f78d
I20240502 14:00:36.836716 18897 raft_consensus.cc:348] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:36.836824 18897 raft_consensus.cc:729] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f688370715454d6e814a78fcf85ac5c3, State: Initialized, Role: LEARNER
I20240502 14:00:36.836937 15832 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.836925 18897 consensus_queue.cc:260] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:36.837287 18447 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.837635 18447 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.839051 18174 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.840595 18858 ts_tablet_manager.cc:1367] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25: Time spent bootstrapping tablet: real 0.161s user 0.030s sys 0.000s
I20240502 14:00:36.830492 17381 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:36.841094 18858 raft_consensus.cc:348] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:36.841218 18858 raft_consensus.cc:729] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c3658c946994e2ba9f41b64aa327d25, State: Initialized, Role: LEARNER
I20240502 14:00:36.841202 18401 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:36.841315 18858 consensus_queue.cc:260] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:36.841626 18858 ts_tablet_manager.cc:1398] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:36.842119 18897 ts_tablet_manager.cc:1398] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:00:36.842260 15721 tablet_copy_service.cc:342] P 8484181c4ece483c8b408817fab1a917: Request end of tablet copy session 3c3658c946994e2ba9f41b64aa327d25-1f80c8590e994a10a9b7c6c174e28173 received from {username='slave'} at 127.8.86.134:44409
I20240502 14:00:36.842346 15721 tablet_copy_service.cc:433] P 8484181c4ece483c8b408817fab1a917: ending tablet copy session 3c3658c946994e2ba9f41b64aa327d25-1f80c8590e994a10a9b7c6c174e28173 on tablet 1f80c8590e994a10a9b7c6c174e28173 with peer 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:36.843781 15590 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session f688370715454d6e814a78fcf85ac5c3-1f562ad00ec34a8d9184671ed55d84d1 received from {username='slave'} at 127.8.86.133:50351
I20240502 14:00:36.843858 15590 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session f688370715454d6e814a78fcf85ac5c3-1f562ad00ec34a8d9184671ed55d84d1 on tablet 1f562ad00ec34a8d9184671ed55d84d1 with peer f688370715454d6e814a78fcf85ac5c3
I20240502 14:00:36.846324 17381 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:36.846501 15964 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.847388 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:36.849191 18605 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.849876 15832 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.850961 15962 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.854434 15420 catalog_manager.cc:5484] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.856233 15962 consensus_queue.cc:237] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:36.858067 15570 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.858556 18418 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.859012 18917 ts_tablet_manager.cc:902] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
W20240502 14:00:36.859308 15898 consensus_peers.cc:485] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 387bdd8ae3b34f588e2170b061a5bc05. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.859669 18917 tablet_copy_client.cc:308] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
I20240502 14:00:36.860285 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 22eef55211ca4f65979022d2f4ae829f from peer 8fea283177394bd28d7eacf4fb744d59 ({username='slave'} at 127.8.86.135:39871)
I20240502 14:00:36.860388 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 22eef55211ca4f65979022d2f4ae829f from peer 8fea283177394bd28d7eacf4fb744d59 at {username='slave'} at 127.8.86.135:39871: session id = 8fea283177394bd28d7eacf4fb744d59-22eef55211ca4f65979022d2f4ae829f
I20240502 14:00:36.860718 18418 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:36.861371 15853 tablet_copy_source_session.cc:205] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.862399 18917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22eef55211ca4f65979022d2f4ae829f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:00:36.863024 15897 consensus_peers.cc:485] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.864158 15570 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:36.864822 15832 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:36.865150 18917 tablet_copy_client.cc:783] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.865211 18566 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:36.865412 18917 tablet_copy_client.cc:647] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.865916 17515 raft_consensus.cc:1203] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:36.866127 15833 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:36.866590 18917 tablet_copy_client.cc:523] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.866624 15420 catalog_manager.cc:5484] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:36.868012 18917 tablet_bootstrap.cc:492] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59: Bootstrap starting.
I20240502 14:00:36.872768 19073 ts_tablet_manager.cc:902] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:36.872922 18926 ts_tablet_manager.cc:902] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:36.873327 19073 tablet_copy_client.cc:308] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:36.873406 18926 tablet_copy_client.cc:308] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:36.873781 15590 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet b0cd21dc897b41aca9118be6df48cc0c from peer 3c3658c946994e2ba9f41b64aa327d25 ({username='slave'} at 127.8.86.134:43305)
I20240502 14:00:36.873871 15590 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet b0cd21dc897b41aca9118be6df48cc0c from peer 3c3658c946994e2ba9f41b64aa327d25 at {username='slave'} at 127.8.86.134:43305: session id = 3c3658c946994e2ba9f41b64aa327d25-b0cd21dc897b41aca9118be6df48cc0c
I20240502 14:00:36.874835 15590 tablet_copy_source_session.cc:205] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.875856 19073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0cd21dc897b41aca9118be6df48cc0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.877882 19073 tablet_copy_client.cc:783] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.878007 19073 tablet_copy_client.cc:647] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.879326 15591 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet bb9f100600a246708457280d91826fce from peer 2254b3ee3db647598d781e467584f78d ({username='slave'} at 127.8.86.136:58333)
I20240502 14:00:36.879426 15591 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet bb9f100600a246708457280d91826fce from peer 2254b3ee3db647598d781e467584f78d at {username='slave'} at 127.8.86.136:58333: session id = 2254b3ee3db647598d781e467584f78d-bb9f100600a246708457280d91826fce
I20240502 14:00:36.879662 19073 tablet_copy_client.cc:523] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.880125 18773 tablet_bootstrap.cc:492] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.880244 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:36.880362 15591 tablet_copy_source_session.cc:205] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.880441 18773 tablet_bootstrap.cc:492] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap complete.
I20240502 14:00:36.880769 18773 ts_tablet_manager.cc:1367] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25: Time spent bootstrapping tablet: real 0.538s user 0.028s sys 0.000s
I20240502 14:00:36.880803 19073 tablet_bootstrap.cc:492] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap starting.
I20240502 14:00:36.881247 18773 raft_consensus.cc:348] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:36.881371 18773 raft_consensus.cc:729] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c3658c946994e2ba9f41b64aa327d25, State: Initialized, Role: LEARNER
I20240502 14:00:36.881580 18773 consensus_queue.cc:260] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:36.882577 18773 ts_tablet_manager.cc:1398] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:36.887738 19001 tablet_copy_client.cc:523] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.890190 19001 tablet_bootstrap.cc:492] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3: Bootstrap starting.
I20240502 14:00:36.893481 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:36.893987 18928 tablet_bootstrap.cc:492] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.894345 18928 tablet_bootstrap.cc:492] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d: Bootstrap complete.
I20240502 14:00:36.894554 18928 ts_tablet_manager.cc:1367] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d: Time spent bootstrapping tablet: real 0.090s user 0.029s sys 0.001s
I20240502 14:00:36.895048 18928 raft_consensus.cc:348] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:36.895272 18928 raft_consensus.cc:729] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2254b3ee3db647598d781e467584f78d, State: Initialized, Role: LEARNER
I20240502 14:00:36.895409 18928 consensus_queue.cc:260] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:36.896353 18928 ts_tablet_manager.cc:1398] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:36.897038 18763 tablet_bootstrap.cc:492] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.897513 18763 tablet_bootstrap.cc:492] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d: Bootstrap complete.
I20240502 14:00:36.897677 18763 ts_tablet_manager.cc:1367] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d: Time spent bootstrapping tablet: real 0.600s user 0.040s sys 0.000s
I20240502 14:00:36.898051 18763 raft_consensus.cc:348] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:36.898161 18763 raft_consensus.cc:729] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2254b3ee3db647598d781e467584f78d, State: Initialized, Role: LEARNER
I20240502 14:00:36.902494 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session 2254b3ee3db647598d781e467584f78d-2886dfbec1e141198b0cf7a435fac498 received from {username='slave'} at 127.8.86.136:53895
I20240502 14:00:36.902585 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session 2254b3ee3db647598d781e467584f78d-2886dfbec1e141198b0cf7a435fac498 on tablet 2886dfbec1e141198b0cf7a435fac498 with peer 2254b3ee3db647598d781e467584f78d
I20240502 14:00:36.911094 17488 tablet_service.cc:1514] Processing DeleteTablet for tablet 436c127af87c49e8b7bf9b6571f7dc30 with delete_type TABLET_DATA_TOMBSTONED (TS 8fea283177394bd28d7eacf4fb744d59 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59474
I20240502 14:00:36.912516 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:36.919296 18788 tablet_bootstrap.cc:492] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.919760 18788 tablet_bootstrap.cc:492] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59: Bootstrap complete.
I20240502 14:00:36.920020 18788 ts_tablet_manager.cc:1367] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59: Time spent bootstrapping tablet: real 0.565s user 0.028s sys 0.000s
I20240502 14:00:36.920558 18788 raft_consensus.cc:348] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:36.920754 18788 raft_consensus.cc:729] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8fea283177394bd28d7eacf4fb744d59, State: Initialized, Role: LEARNER
I20240502 14:00:36.920902 18788 consensus_queue.cc:260] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:36.916864 19073 tablet_bootstrap.cc:492] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.923764 19073 tablet_bootstrap.cc:492] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap complete.
I20240502 14:00:36.923995 19073 ts_tablet_manager.cc:1367] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25: Time spent bootstrapping tablet: real 0.043s user 0.024s sys 0.000s
I20240502 14:00:36.924522 19073 raft_consensus.cc:348] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:36.924710 19073 raft_consensus.cc:729] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c3658c946994e2ba9f41b64aa327d25, State: Initialized, Role: LEARNER
I20240502 14:00:36.924851 19073 consensus_queue.cc:260] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:36.926156 19073 ts_tablet_manager.cc:1398] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:36.927371 15591 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session 3c3658c946994e2ba9f41b64aa327d25-619a8ec66ba74ecea689c17b4734253a received from {username='slave'} at 127.8.86.134:43305
I20240502 14:00:36.927460 15591 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session 3c3658c946994e2ba9f41b64aa327d25-619a8ec66ba74ecea689c17b4734253a on tablet 619a8ec66ba74ecea689c17b4734253a with peer 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:36.929749 15591 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session 3c3658c946994e2ba9f41b64aa327d25-b0cd21dc897b41aca9118be6df48cc0c received from {username='slave'} at 127.8.86.134:43305
I20240502 14:00:36.929827 15591 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session 3c3658c946994e2ba9f41b64aa327d25-b0cd21dc897b41aca9118be6df48cc0c on tablet b0cd21dc897b41aca9118be6df48cc0c with peer 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:36.931509 17515 raft_consensus.cc:1203] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:36.933413 18897 ts_tablet_manager.cc:902] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:36.933928 18897 tablet_copy_client.cc:308] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
I20240502 14:00:36.934695 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 2886dfbec1e141198b0cf7a435fac498 from peer f688370715454d6e814a78fcf85ac5c3 ({username='slave'} at 127.8.86.133:37489)
I20240502 14:00:36.934800 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 2886dfbec1e141198b0cf7a435fac498 from peer f688370715454d6e814a78fcf85ac5c3 at {username='slave'} at 127.8.86.133:37489: session id = f688370715454d6e814a78fcf85ac5c3-2886dfbec1e141198b0cf7a435fac498
I20240502 14:00:36.935715 15853 tablet_copy_source_session.cc:205] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.936590 18897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2886dfbec1e141198b0cf7a435fac498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.938458 18763 consensus_queue.cc:260] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:36.938809 18897 tablet_copy_client.cc:783] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.938956 18897 tablet_copy_client.cc:647] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.938944 18763 ts_tablet_manager.cc:1398] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d: Time spent starting tablet: real 0.041s user 0.001s sys 0.000s
I20240502 14:00:36.939435 15721 tablet_copy_service.cc:342] P 8484181c4ece483c8b408817fab1a917: Request end of tablet copy session 2254b3ee3db647598d781e467584f78d-76f07464d9bd49a9b1b9d58f6a180cf8 received from {username='slave'} at 127.8.86.136:42777
I20240502 14:00:36.918856 18926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9f100600a246708457280d91826fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.939538 15721 tablet_copy_service.cc:433] P 8484181c4ece483c8b408817fab1a917: ending tablet copy session 2254b3ee3db647598d781e467584f78d-76f07464d9bd49a9b1b9d58f6a180cf8 on tablet 76f07464d9bd49a9b1b9d58f6a180cf8 with peer 2254b3ee3db647598d781e467584f78d
I20240502 14:00:36.940241 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:36.940938 18447 raft_consensus.cc:1050] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917: attempting to promote NON_VOTER f688370715454d6e814a78fcf85ac5c3 to VOTER
I20240502 14:00:36.941404 18447 consensus_queue.cc:237] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:36.942226 18926 tablet_copy_client.cc:783] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.943984 17250 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.944690 15832 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:36.945044 18174 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:36.946098 17251 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:36.947425 18174 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:36.948053 18897 tablet_copy_client.cc:523] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.949251 15962 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:00:36.949306 15635 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48fbc670ec504d26a11bf1b9fbd78981. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:36.949342 18897 tablet_bootstrap.cc:492] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3: Bootstrap starting.
I20240502 14:00:36.950403 18502 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:36.953300 18502 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.953600 18751 tablet_bootstrap.cc:492] T 9870bc797f184a6284ff6ca4f78b3826 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.953616 15962 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.953861 18751 tablet_bootstrap.cc:492] T 9870bc797f184a6284ff6ca4f78b3826 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap complete.
I20240502 14:00:36.954025 18751 ts_tablet_manager.cc:1367] T 9870bc797f184a6284ff6ca4f78b3826 P 3c3658c946994e2ba9f41b64aa327d25: Time spent bootstrapping tablet: real 0.234s user 0.028s sys 0.000s
I20240502 14:00:36.954412 19073 ts_tablet_manager.cc:902] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:36.954540 18751 raft_consensus.cc:348] T 9870bc797f184a6284ff6ca4f78b3826 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:36.954656 18751 raft_consensus.cc:729] T 9870bc797f184a6284ff6ca4f78b3826 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c3658c946994e2ba9f41b64aa327d25, State: Initialized, Role: LEARNER
I20240502 14:00:36.954764 18751 consensus_queue.cc:260] T 9870bc797f184a6284ff6ca4f78b3826 P 3c3658c946994e2ba9f41b64aa327d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:36.954879 19073 tablet_copy_client.cc:308] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:36.957082 15422 catalog_manager.cc:5484] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.958020 15591 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet dfdcf1124e8a474facc14c45e8963568 from peer 3c3658c946994e2ba9f41b64aa327d25 ({username='slave'} at 127.8.86.134:43305)
I20240502 14:00:36.958132 15591 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet dfdcf1124e8a474facc14c45e8963568 from peer 3c3658c946994e2ba9f41b64aa327d25 at {username='slave'} at 127.8.86.134:43305: session id = 3c3658c946994e2ba9f41b64aa327d25-dfdcf1124e8a474facc14c45e8963568
I20240502 14:00:36.959074 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:36.960064 18821 ts_tablet_manager.cc:902] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:36.960192 15591 tablet_copy_source_session.cc:205] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.960588 18821 tablet_copy_client.cc:308] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:36.961621 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:36.962030 15591 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet dfdcf1124e8a474facc14c45e8963568 from peer 8fea283177394bd28d7eacf4fb744d59 ({username='slave'} at 127.8.86.135:41945)
I20240502 14:00:36.962121 15591 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet dfdcf1124e8a474facc14c45e8963568 from peer 8fea283177394bd28d7eacf4fb744d59 at {username='slave'} at 127.8.86.135:41945: session id = 8fea283177394bd28d7eacf4fb744d59-dfdcf1124e8a474facc14c45e8963568
I20240502 14:00:36.962903 15591 tablet_copy_source_session.cc:205] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.966825 18821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfdcf1124e8a474facc14c45e8963568. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.967067 17249 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.968688 17248 raft_consensus.cc:1203] T b07f4db6f82e4a56accdbf1bbbc5f965 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:36.969012 18821 tablet_copy_client.cc:783] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.969167 18821 tablet_copy_client.cc:647] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.969681 17250 raft_consensus.cc:1203] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:00:36.970618 18841 ts_tablet_manager.cc:902] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:36.971174 18841 tablet_copy_client.cc:308] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:36.972591 15591 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet dfdcf1124e8a474facc14c45e8963568 from peer f688370715454d6e814a78fcf85ac5c3 ({username='slave'} at 127.8.86.133:50351)
I20240502 14:00:36.972715 15591 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet dfdcf1124e8a474facc14c45e8963568 from peer f688370715454d6e814a78fcf85ac5c3 at {username='slave'} at 127.8.86.133:50351: session id = f688370715454d6e814a78fcf85ac5c3-dfdcf1124e8a474facc14c45e8963568
I20240502 14:00:36.972576 15832 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:36.973841 15591 tablet_copy_source_session.cc:205] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.974756 18841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfdcf1124e8a474facc14c45e8963568. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.974943 19073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfdcf1124e8a474facc14c45e8963568. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.976956 18841 tablet_copy_client.cc:783] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.977092 18841 tablet_copy_client.cc:647] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.978148 18926 tablet_copy_client.cc:647] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.978549 18977 tablet_bootstrap.cc:492] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:36.979030 15700 consensus_queue.cc:237] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:36.979489 19073 tablet_copy_client.cc:783] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.979643 19073 tablet_copy_client.cc:647] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.981698 15962 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:36.982604 18605 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:36.983289 15832 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:36.983451 17251 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:36.983685 18605 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:36.984236 18605 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:00:36.985819 15415 catalog_manager.cc:4797] TS 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): delete failed for tablet 436c127af87c49e8b7bf9b6571f7dc30 because the tablet was not found. No further retry: Not found: Tablet not found: 436c127af87c49e8b7bf9b6571f7dc30
I20240502 14:00:36.986301 18605 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:36.988523 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 48fbc670ec504d26a11bf1b9fbd78981 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:36.988765 18841 tablet_copy_client.cc:523] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:36.988701 15962 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:36.988857 15832 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:36.989034 15422 catalog_manager.cc:5484] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:36.990051 18841 tablet_bootstrap.cc:492] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3: Bootstrap starting.
I20240502 14:00:36.991416 17515 raft_consensus.cc:1203] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:00:36.993254 18832 ts_tablet_manager.cc:902] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:36.993772 18832 tablet_copy_client.cc:308] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
I20240502 14:00:36.994747 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 02d3da05d82340178e27218c2ce56b67 from peer f688370715454d6e814a78fcf85ac5c3 ({username='slave'} at 127.8.86.133:37489)
I20240502 14:00:36.994858 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 02d3da05d82340178e27218c2ce56b67 from peer f688370715454d6e814a78fcf85ac5c3 at {username='slave'} at 127.8.86.133:37489: session id = f688370715454d6e814a78fcf85ac5c3-02d3da05d82340178e27218c2ce56b67
I20240502 14:00:36.995793 15853 tablet_copy_source_session.cc:205] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:36.996577 18832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d3da05d82340178e27218c2ce56b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.998582 18832 tablet_copy_client.cc:783] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:36.998716 18832 tablet_copy_client.cc:647] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:36.998746 18977 tablet_bootstrap.cc:492] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d: Bootstrap complete.
I20240502 14:00:36.998939 18977 ts_tablet_manager.cc:1367] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d: Time spent bootstrapping tablet: real 0.262s user 0.024s sys 0.005s
I20240502 14:00:36.999461 18977 raft_consensus.cc:348] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:36.999702 18977 raft_consensus.cc:729] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2254b3ee3db647598d781e467584f78d, State: Initialized, Role: LEARNER
I20240502 14:00:36.999881 18977 consensus_queue.cc:260] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.000365 18977 ts_tablet_manager.cc:1398] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.000242 17251 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:36.985141 18926 tablet_copy_client.cc:523] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:37.000855 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session 2254b3ee3db647598d781e467584f78d-22eef55211ca4f65979022d2f4ae829f received from {username='slave'} at 127.8.86.136:53895
I20240502 14:00:37.000929 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session 2254b3ee3db647598d781e467584f78d-22eef55211ca4f65979022d2f4ae829f on tablet 22eef55211ca4f65979022d2f4ae829f with peer 2254b3ee3db647598d781e467584f78d
I20240502 14:00:37.002264 18926 tablet_bootstrap.cc:492] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d: Bootstrap starting.
I20240502 14:00:37.004161 17381 raft_consensus.cc:1203] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:36.955050 18751 ts_tablet_manager.cc:1398] T 9870bc797f184a6284ff6ca4f78b3826 P 3c3658c946994e2ba9f41b64aa327d25: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.006685 17647 raft_consensus.cc:1203] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:37.007090 18832 tablet_copy_client.cc:523] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:37.008337 18832 tablet_bootstrap.cc:492] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3: Bootstrap starting.
I20240502 14:00:37.009263 15591 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session 3c3658c946994e2ba9f41b64aa327d25-9870bc797f184a6284ff6ca4f78b3826 received from {username='slave'} at 127.8.86.134:43305
I20240502 14:00:37.009341 15591 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session 3c3658c946994e2ba9f41b64aa327d25-9870bc797f184a6284ff6ca4f78b3826 on tablet 9870bc797f184a6284ff6ca4f78b3826 with peer 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:37.010496 18821 tablet_copy_client.cc:523] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:37.011968 18821 tablet_bootstrap.cc:492] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap starting.
I20240502 14:00:37.016645 15700 consensus_queue.cc:237] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:37.018229 17251 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.023047 17362 tablet_service.cc:1514] Processing DeleteTablet for tablet 48fbc670ec504d26a11bf1b9fbd78981 with delete_type TABLET_DATA_TOMBSTONED (TS 3c3658c946994e2ba9f41b64aa327d25 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37172
I20240502 14:00:37.023133 18174 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:37.024019 19121 ts_tablet_manager.cc:902] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59: Initiating tablet copy from peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:37.025190 19121 tablet_copy_client.cc:308] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:42261
I20240502 14:00:37.025173 18174 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.026610 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 48fbc670ec504d26a11bf1b9fbd78981 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.028638 17251 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.031322 18700 tablet_bootstrap.cc:492] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:37.031450 15425 catalog_manager.cc:5484] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3 reported cstate change: config changed from index 7 to 8, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.032338 18977 ts_tablet_manager.cc:902] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:37.032822 18977 tablet_copy_client.cc:308] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
I20240502 14:00:37.033356 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 1c004727e7c64f33bc9492184b77fde9 from peer 2254b3ee3db647598d781e467584f78d ({username='slave'} at 127.8.86.136:53895)
I20240502 14:00:37.033469 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 1c004727e7c64f33bc9492184b77fde9 from peer 2254b3ee3db647598d781e467584f78d at {username='slave'} at 127.8.86.136:53895: session id = 2254b3ee3db647598d781e467584f78d-1c004727e7c64f33bc9492184b77fde9
I20240502 14:00:37.034423 15853 tablet_copy_source_session.cc:205] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:37.035467 15944 tablet_service.cc:1514] Processing DeleteTablet for tablet 48fbc670ec504d26a11bf1b9fbd78981 with delete_type TABLET_DATA_TOMBSTONED (TS 4d0f8f311e834d81bd54b9f8e7923bcd not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:56876
I20240502 14:00:37.038705 18977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c004727e7c64f33bc9492184b77fde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.039037 19134 tablet_replica.cc:330] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: stopping tablet replica
I20240502 14:00:37.039417 19134 raft_consensus.cc:2229] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.039741 19134 raft_consensus.cc:2258] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.041496 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:37.041754 19134 ts_tablet_manager.cc:1875] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.041997 18977 tablet_copy_client.cc:783] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:37.042151 18977 tablet_copy_client.cc:647] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:37.042380 15832 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.042797 18605 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:37.044123 15832 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.051520 19134 ts_tablet_manager.cc:1888] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:00:37.051699 19134 log.cc:1196] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal/wals/48fbc670ec504d26a11bf1b9fbd78981
I20240502 14:00:37.052456 15413 catalog_manager.cc:4830] TS 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719): tablet 48fbc670ec504d26a11bf1b9fbd78981 (table rebalance_test_table_2 [id=b36f1f16cb3d4b8bae9303e2976dc2e7]) successfully deleted
I20240502 14:00:37.053568 18977 tablet_copy_client.cc:523] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:37.054616 17381 raft_consensus.cc:1203] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:00:37.054915 18977 tablet_bootstrap.cc:492] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d: Bootstrap starting.
I20240502 14:00:37.057104 18897 tablet_bootstrap.cc:492] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:37.057410 18897 tablet_bootstrap.cc:492] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3: Bootstrap complete.
I20240502 14:00:37.057930 18897 ts_tablet_manager.cc:1367] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3: Time spent bootstrapping tablet: real 0.109s user 0.021s sys 0.007s
I20240502 14:00:37.057766 17647 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.058264 15721 tablet_copy_service.cc:140] P 8484181c4ece483c8b408817fab1a917: Received BeginTabletCopySession request for tablet a7981c3f46d8483caae82fbf2296e762 from peer 8fea283177394bd28d7eacf4fb744d59 ({username='slave'} at 127.8.86.135:39313)
I20240502 14:00:37.058405 15721 tablet_copy_service.cc:161] P 8484181c4ece483c8b408817fab1a917: Beginning new tablet copy session on tablet a7981c3f46d8483caae82fbf2296e762 from peer 8fea283177394bd28d7eacf4fb744d59 at {username='slave'} at 127.8.86.135:39313: session id = 8fea283177394bd28d7eacf4fb744d59-a7981c3f46d8483caae82fbf2296e762
I20240502 14:00:37.059355 15721 tablet_copy_source_session.cc:205] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:37.059299 18897 raft_consensus.cc:348] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:37.059420 18897 raft_consensus.cc:729] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f688370715454d6e814a78fcf85ac5c3, State: Initialized, Role: LEARNER
I20240502 14:00:37.059515 18897 consensus_queue.cc:260] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:37.060266 19121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7981c3f46d8483caae82fbf2296e762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.061046 18897 ts_tablet_manager.cc:1398] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:37.062839 19121 tablet_copy_client.cc:783] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:37.062987 19121 tablet_copy_client.cc:647] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:37.063273 18700 tablet_bootstrap.cc:492] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap complete.
I20240502 14:00:37.063448 18700 ts_tablet_manager.cc:1367] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59: Time spent bootstrapping tablet: real 0.285s user 0.028s sys 0.000s
I20240502 14:00:37.064167 19121 tablet_copy_client.cc:523] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:37.064302 18700 raft_consensus.cc:348] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.064401 18700 raft_consensus.cc:729] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8fea283177394bd28d7eacf4fb744d59, State: Initialized, Role: LEARNER
I20240502 14:00:37.064554 18700 consensus_queue.cc:260] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.065320 19121 tablet_bootstrap.cc:492] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap starting.
I20240502 14:00:37.065402 18700 ts_tablet_manager.cc:1398] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:37.065881 15984 tablet_copy_service.cc:342] P 4d0f8f311e834d81bd54b9f8e7923bcd: Request end of tablet copy session 8fea283177394bd28d7eacf4fb744d59-2d92e5885bf94c3e8e0e4dee40017699 received from {username='slave'} at 127.8.86.135:44181
I20240502 14:00:37.065953 15984 tablet_copy_service.cc:433] P 4d0f8f311e834d81bd54b9f8e7923bcd: ending tablet copy session 8fea283177394bd28d7eacf4fb744d59-2d92e5885bf94c3e8e0e4dee40017699 on tablet 2d92e5885bf94c3e8e0e4dee40017699 with peer 8fea283177394bd28d7eacf4fb744d59
I20240502 14:00:37.065946 19073 tablet_copy_client.cc:523] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:37.067274 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session f688370715454d6e814a78fcf85ac5c3-2886dfbec1e141198b0cf7a435fac498 received from {username='slave'} at 127.8.86.133:37489
I20240502 14:00:37.067344 19073 tablet_bootstrap.cc:492] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap starting.
I20240502 14:00:37.067377 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session f688370715454d6e814a78fcf85ac5c3-2886dfbec1e141198b0cf7a435fac498 on tablet 2886dfbec1e141198b0cf7a435fac498 with peer f688370715454d6e814a78fcf85ac5c3
I20240502 14:00:37.073980 18897 ts_tablet_manager.cc:902] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3: Initiating tablet copy from peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:37.074256 18788 ts_tablet_manager.cc:1398] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59: Time spent starting tablet: real 0.154s user 0.001s sys 0.000s
I20240502 14:00:37.074621 18897 tablet_copy_client.cc:308] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.132:45719
I20240502 14:00:37.075493 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session 8fea283177394bd28d7eacf4fb744d59-03dd735113ab4ea6bdb696b11a4efd7f received from {username='slave'} at 127.8.86.135:39871
I20240502 14:00:37.075580 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session 8fea283177394bd28d7eacf4fb744d59-03dd735113ab4ea6bdb696b11a4efd7f on tablet 03dd735113ab4ea6bdb696b11a4efd7f with peer 8fea283177394bd28d7eacf4fb744d59
I20240502 14:00:37.078936 17647 raft_consensus.cc:1203] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:00:37.080569 15984 tablet_copy_service.cc:140] P 4d0f8f311e834d81bd54b9f8e7923bcd: Received BeginTabletCopySession request for tablet 837e8b1235a44131bdaef24b763d1cac from peer f688370715454d6e814a78fcf85ac5c3 ({username='slave'} at 127.8.86.133:49061)
I20240502 14:00:37.080706 15984 tablet_copy_service.cc:161] P 4d0f8f311e834d81bd54b9f8e7923bcd: Beginning new tablet copy session on tablet 837e8b1235a44131bdaef24b763d1cac from peer f688370715454d6e814a78fcf85ac5c3 at {username='slave'} at 127.8.86.133:49061: session id = f688370715454d6e814a78fcf85ac5c3-837e8b1235a44131bdaef24b763d1cac
I20240502 14:00:37.081607 15984 tablet_copy_source_session.cc:205] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:37.084728 18897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837e8b1235a44131bdaef24b763d1cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.094347 17647 raft_consensus.cc:1203] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:00:37.095455 18897 tablet_copy_client.cc:783] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:37.095616 18897 tablet_copy_client.cc:647] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:37.111418 18897 tablet_copy_client.cc:523] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:37.112897 18897 tablet_bootstrap.cc:492] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3: Bootstrap starting.
I20240502 14:00:37.115687 15570 consensus_queue.cc:237] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:37.115963 18821 tablet_bootstrap.cc:492] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:37.117928 15832 raft_consensus.cc:1261] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.118372 18821 tablet_bootstrap.cc:492] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap complete.
I20240502 14:00:37.118654 18821 ts_tablet_manager.cc:1367] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59: Time spent bootstrapping tablet: real 0.107s user 0.026s sys 0.002s
I20240502 14:00:37.118671 17167 consensus_queue.cc:1035] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.119217 18821 raft_consensus.cc:348] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.119426 18821 raft_consensus.cc:729] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8fea283177394bd28d7eacf4fb744d59, State: Initialized, Role: LEARNER
I20240502 14:00:37.119602 18821 consensus_queue.cc:260] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.119657 17381 raft_consensus.cc:1261] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.121029 18821 ts_tablet_manager.cc:1398] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:37.121336 19048 consensus_queue.cc:1035] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.121829 15591 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session 8fea283177394bd28d7eacf4fb744d59-dfdcf1124e8a474facc14c45e8963568 received from {username='slave'} at 127.8.86.135:41945
I20240502 14:00:37.121904 15591 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session 8fea283177394bd28d7eacf4fb744d59-dfdcf1124e8a474facc14c45e8963568 on tablet dfdcf1124e8a474facc14c45e8963568 with peer 8fea283177394bd28d7eacf4fb744d59
I20240502 14:00:37.121973 15700 raft_consensus.cc:1261] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.123589 19048 consensus_queue.cc:1035] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:00:37.125327 15505 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 186a946321e445b58b0097c658847ebe. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.133554 18751 ts_tablet_manager.cc:902] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25: Initiating tablet copy from peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:37.134096 18751 tablet_copy_client.cc:308] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:42261
I20240502 14:00:37.134953 15721 tablet_copy_service.cc:140] P 8484181c4ece483c8b408817fab1a917: Received BeginTabletCopySession request for tablet 76f07464d9bd49a9b1b9d58f6a180cf8 from peer 3c3658c946994e2ba9f41b64aa327d25 ({username='slave'} at 127.8.86.134:44409)
I20240502 14:00:37.135054 15721 tablet_copy_service.cc:161] P 8484181c4ece483c8b408817fab1a917: Beginning new tablet copy session on tablet 76f07464d9bd49a9b1b9d58f6a180cf8 from peer 3c3658c946994e2ba9f41b64aa327d25 at {username='slave'} at 127.8.86.134:44409: session id = 3c3658c946994e2ba9f41b64aa327d25-76f07464d9bd49a9b1b9d58f6a180cf8
I20240502 14:00:37.135977 15721 tablet_copy_source_session.cc:205] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:37.137084 18751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f07464d9bd49a9b1b9d58f6a180cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.138346 15833 consensus_queue.cc:237] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:37.139564 18751 tablet_copy_client.cc:783] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:37.139778 18751 tablet_copy_client.cc:647] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:37.141738 15700 raft_consensus.cc:1261] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:00:37.142107 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.143105 17647 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
W20240502 14:00:37.144661 15766 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c57cab68b764db0917ed98e74757ff0. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.144933 18234 consensus_queue.cc:1035] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.147086 15962 raft_consensus.cc:1261] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.147459 18751 tablet_copy_client.cc:523] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:37.147593 18234 consensus_queue.cc:1035] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.148988 18751 tablet_bootstrap.cc:492] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap starting.
I20240502 14:00:37.152614 19073 tablet_bootstrap.cc:492] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:37.152920 18841 tablet_bootstrap.cc:492] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:37.154773 19073 tablet_bootstrap.cc:492] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap complete.
I20240502 14:00:37.155016 19073 ts_tablet_manager.cc:1367] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25: Time spent bootstrapping tablet: real 0.088s user 0.029s sys 0.000s
I20240502 14:00:37.155519 19073 raft_consensus.cc:348] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.155732 19073 raft_consensus.cc:729] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c3658c946994e2ba9f41b64aa327d25, State: Initialized, Role: LEARNER
I20240502 14:00:37.155937 15570 consensus_queue.cc:237] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:37.156286 18841 tablet_bootstrap.cc:492] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3: Bootstrap complete.
I20240502 14:00:37.156232 19073 consensus_queue.cc:260] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.157274 18841 ts_tablet_manager.cc:1367] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3: Time spent bootstrapping tablet: real 0.167s user 0.020s sys 0.010s
I20240502 14:00:37.157764 18841 raft_consensus.cc:348] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.158140 17515 raft_consensus.cc:1261] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.158748 19073 ts_tablet_manager.cc:1398] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:37.158779 18402 consensus_queue.cc:1035] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:00:37.159166 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d81f14b7c1d1468196d0fe6d67fe93a7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.159633 15702 raft_consensus.cc:1261] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.159996 15591 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session 3c3658c946994e2ba9f41b64aa327d25-dfdcf1124e8a474facc14c45e8963568 received from {username='slave'} at 127.8.86.134:43305
I20240502 14:00:37.160061 15591 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session 3c3658c946994e2ba9f41b64aa327d25-dfdcf1124e8a474facc14c45e8963568 on tablet dfdcf1124e8a474facc14c45e8963568 with peer 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:37.160809 18402 consensus_queue.cc:1035] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.161356 19121 tablet_bootstrap.cc:492] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:37.161710 19121 tablet_bootstrap.cc:492] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap complete.
I20240502 14:00:37.161916 19121 ts_tablet_manager.cc:1367] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59: Time spent bootstrapping tablet: real 0.097s user 0.024s sys 0.005s
I20240502 14:00:37.162387 19121 raft_consensus.cc:348] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.162617 19121 raft_consensus.cc:729] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8fea283177394bd28d7eacf4fb744d59, State: Initialized, Role: LEARNER
I20240502 14:00:37.162760 19121 consensus_queue.cc:260] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.162883 15833 raft_consensus.cc:1261] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.163347 18402 consensus_queue.cc:1035] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.163375 18841 raft_consensus.cc:729] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f688370715454d6e814a78fcf85ac5c3, State: Initialized, Role: LEARNER
I20240502 14:00:37.163532 18841 consensus_queue.cc:260] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.163892 19121 ts_tablet_manager.cc:1398] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:37.164347 15721 tablet_copy_service.cc:342] P 8484181c4ece483c8b408817fab1a917: Request end of tablet copy session 8fea283177394bd28d7eacf4fb744d59-a7981c3f46d8483caae82fbf2296e762 received from {username='slave'} at 127.8.86.135:39313
I20240502 14:00:37.164433 15721 tablet_copy_service.cc:433] P 8484181c4ece483c8b408817fab1a917: ending tablet copy session 8fea283177394bd28d7eacf4fb744d59-a7981c3f46d8483caae82fbf2296e762 on tablet a7981c3f46d8483caae82fbf2296e762 with peer 8fea283177394bd28d7eacf4fb744d59
I20240502 14:00:37.164774 18841 ts_tablet_manager.cc:1398] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 14:00:37.165254 15591 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session f688370715454d6e814a78fcf85ac5c3-dfdcf1124e8a474facc14c45e8963568 received from {username='slave'} at 127.8.86.133:50351
I20240502 14:00:37.165319 15591 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session f688370715454d6e814a78fcf85ac5c3-dfdcf1124e8a474facc14c45e8963568 on tablet dfdcf1124e8a474facc14c45e8963568 with peer f688370715454d6e814a78fcf85ac5c3
I20240502 14:00:37.167578 18832 tablet_bootstrap.cc:492] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:37.168007 18832 tablet_bootstrap.cc:492] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3: Bootstrap complete.
I20240502 14:00:37.168383 18832 ts_tablet_manager.cc:1367] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3: Time spent bootstrapping tablet: real 0.160s user 0.027s sys 0.002s
I20240502 14:00:37.168761 18402 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.169497 18832 raft_consensus.cc:348] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:37.170781 18832 raft_consensus.cc:729] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f688370715454d6e814a78fcf85ac5c3, State: Initialized, Role: LEARNER
I20240502 14:00:37.170953 18832 consensus_queue.cc:260] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:37.171763 18832 ts_tablet_manager.cc:1398] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:37.172345 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session f688370715454d6e814a78fcf85ac5c3-02d3da05d82340178e27218c2ce56b67 received from {username='slave'} at 127.8.86.133:37489
I20240502 14:00:37.172423 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session f688370715454d6e814a78fcf85ac5c3-02d3da05d82340178e27218c2ce56b67 on tablet 02d3da05d82340178e27218c2ce56b67 with peer f688370715454d6e814a78fcf85ac5c3
I20240502 14:00:37.173144 15833 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.174124 15422 catalog_manager.cc:5484] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.175666 18400 raft_consensus.cc:1050] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf: attempting to promote NON_VOTER 8fea283177394bd28d7eacf4fb744d59 to VOTER
I20240502 14:00:37.176048 18400 consensus_queue.cc:237] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:37.177935 17515 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
W20240502 14:00:37.178340 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d81f14b7c1d1468196d0fe6d67fe93a7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.178898 18234 raft_consensus.cc:2932] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.179286 15964 raft_consensus.cc:2932] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.179702 17514 raft_consensus.cc:1261] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.180712 18400 consensus_queue.cc:1035] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.180840 15701 raft_consensus.cc:2932] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.181556 15833 raft_consensus.cc:1261] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.181994 18400 consensus_queue.cc:1035] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.183530 15421 catalog_manager.cc:5484] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.188468 18926 tablet_bootstrap.cc:492] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:37.188328 18400 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.188800 17514 raft_consensus.cc:1203] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:37.190349 17514 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.191732 18926 tablet_bootstrap.cc:492] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d: Bootstrap complete.
I20240502 14:00:37.191908 18926 ts_tablet_manager.cc:1367] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d: Time spent bootstrapping tablet: real 0.190s user 0.021s sys 0.008s
I20240502 14:00:37.191907 15833 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.192356 18926 raft_consensus.cc:348] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:37.192588 18926 raft_consensus.cc:729] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2254b3ee3db647598d781e467584f78d, State: Initialized, Role: LEARNER
I20240502 14:00:37.192749 18926 consensus_queue.cc:260] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:37.193149 18926 ts_tablet_manager.cc:1398] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.194156 18751 tablet_bootstrap.cc:492] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:37.194229 15591 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session 2254b3ee3db647598d781e467584f78d-bb9f100600a246708457280d91826fce received from {username='slave'} at 127.8.86.136:58333
I20240502 14:00:37.194412 15591 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session 2254b3ee3db647598d781e467584f78d-bb9f100600a246708457280d91826fce on tablet bb9f100600a246708457280d91826fce with peer 2254b3ee3db647598d781e467584f78d
W20240502 14:00:37.197769 15413 catalog_manager.cc:4797] TS 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): delete failed for tablet 48fbc670ec504d26a11bf1b9fbd78981 because the tablet was not found. No further retry: Not found: Tablet not found: 48fbc670ec504d26a11bf1b9fbd78981
I20240502 14:00:37.198012 18977 tablet_bootstrap.cc:492] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:37.198333 18977 tablet_bootstrap.cc:492] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d: Bootstrap complete.
I20240502 14:00:37.198518 18977 ts_tablet_manager.cc:1367] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d: Time spent bootstrapping tablet: real 0.144s user 0.029s sys 0.000s
I20240502 14:00:37.199021 18977 raft_consensus.cc:348] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:37.199209 18977 raft_consensus.cc:729] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2254b3ee3db647598d781e467584f78d, State: Initialized, Role: LEARNER
I20240502 14:00:37.199375 18977 consensus_queue.cc:260] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:37.199793 18977 ts_tablet_manager.cc:1398] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.199673 15420 catalog_manager.cc:5484] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.200302 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session 2254b3ee3db647598d781e467584f78d-1c004727e7c64f33bc9492184b77fde9 received from {username='slave'} at 127.8.86.136:53895
I20240502 14:00:37.200388 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session 2254b3ee3db647598d781e467584f78d-1c004727e7c64f33bc9492184b77fde9 on tablet 1c004727e7c64f33bc9492184b77fde9 with peer 2254b3ee3db647598d781e467584f78d
I20240502 14:00:37.205657 19073 ts_tablet_manager.cc:902] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:37.206081 19073 tablet_copy_client.cc:308] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
I20240502 14:00:37.208999 15964 consensus_queue.cc:237] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.210691 15833 consensus_queue.cc:237] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.210809 15831 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.211385 18653 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.213913 15701 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.214480 18751 tablet_bootstrap.cc:492] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap complete.
I20240502 14:00:37.214718 18751 ts_tablet_manager.cc:1367] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25: Time spent bootstrapping tablet: real 0.066s user 0.028s sys 0.002s
I20240502 14:00:37.214794 17646 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.215237 18751 raft_consensus.cc:348] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:37.215540 18751 raft_consensus.cc:729] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c3658c946994e2ba9f41b64aa327d25, State: Initialized, Role: LEARNER
I20240502 14:00:37.215655 18751 consensus_queue.cc:260] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:37.215889 15570 raft_consensus.cc:1261] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:37.216207 15833 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.216475 18751 ts_tablet_manager.cc:1398] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:37.216922 15721 tablet_copy_service.cc:342] P 8484181c4ece483c8b408817fab1a917: Request end of tablet copy session 3c3658c946994e2ba9f41b64aa327d25-76f07464d9bd49a9b1b9d58f6a180cf8 received from {username='slave'} at 127.8.86.134:44409
I20240502 14:00:37.217001 15721 tablet_copy_service.cc:433] P 8484181c4ece483c8b408817fab1a917: ending tablet copy session 3c3658c946994e2ba9f41b64aa327d25-76f07464d9bd49a9b1b9d58f6a180cf8 on tablet 76f07464d9bd49a9b1b9d58f6a180cf8 with peer 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:37.217898 18400 raft_consensus.cc:1050] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf: attempting to promote NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 to VOTER
I20240502 14:00:37.218259 18400 consensus_queue.cc:237] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:37.218981 18584 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:00:37.219380 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28e371c1a999431499e09479e8f5b44b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.220041 15425 catalog_manager.cc:5484] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.221153 18584 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.222757 15831 raft_consensus.cc:1261] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.223066 15570 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.223426 19169 consensus_queue.cc:1035] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:00:37.223851 15505 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 186a946321e445b58b0097c658847ebe. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:37.224056 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e48dda3e62e14e46b03d79648e3e2773. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.224398 18401 consensus_queue.cc:1035] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.225354 18584 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.225605 17381 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.227301 15964 raft_consensus.cc:1261] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:37.227710 19169 consensus_queue.cc:1035] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:37.227775 17382 raft_consensus.cc:1261] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.228407 18402 consensus_queue.cc:1035] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.228593 15701 raft_consensus.cc:1261] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.228935 18402 consensus_queue.cc:1035] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.232498 18653 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.234153 18852 tablet_bootstrap.cc:492] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:37.234912 17382 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.235743 18852 tablet_bootstrap.cc:492] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap complete.
I20240502 14:00:37.235932 18852 ts_tablet_manager.cc:1367] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25: Time spent bootstrapping tablet: real 0.651s user 0.027s sys 0.003s
I20240502 14:00:37.236434 15701 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.237037 15831 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.238214 15420 catalog_manager.cc:5484] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.239557 18852 raft_consensus.cc:348] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:37.239743 18852 raft_consensus.cc:729] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c3658c946994e2ba9f41b64aa327d25, State: Initialized, Role: LEARNER
I20240502 14:00:37.239895 18852 consensus_queue.cc:260] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:37.241168 18852 ts_tablet_manager.cc:1398] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:37.241345 18584 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.241657 15984 tablet_copy_service.cc:342] P 4d0f8f311e834d81bd54b9f8e7923bcd: Request end of tablet copy session 3c3658c946994e2ba9f41b64aa327d25-387bdd8ae3b34f588e2170b061a5bc05 received from {username='slave'} at 127.8.86.134:37349
I20240502 14:00:37.241724 15984 tablet_copy_service.cc:433] P 4d0f8f311e834d81bd54b9f8e7923bcd: ending tablet copy session 3c3658c946994e2ba9f41b64aa327d25-387bdd8ae3b34f588e2170b061a5bc05 on tablet 387bdd8ae3b34f588e2170b061a5bc05 with peer 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:37.242499 17382 raft_consensus.cc:1203] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:37.244120 15569 consensus_queue.cc:237] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }
I20240502 14:00:37.244676 15833 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.247213 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 03dd735113ab4ea6bdb696b11a4efd7f from peer 3c3658c946994e2ba9f41b64aa327d25 ({username='slave'} at 127.8.86.134:56803)
I20240502 14:00:37.247310 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 03dd735113ab4ea6bdb696b11a4efd7f from peer 3c3658c946994e2ba9f41b64aa327d25 at {username='slave'} at 127.8.86.134:56803: session id = 3c3658c946994e2ba9f41b64aa327d25-03dd735113ab4ea6bdb696b11a4efd7f
I20240502 14:00:37.247485 15426 catalog_manager.cc:5484] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 4 to 5, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.248123 15853 tablet_copy_source_session.cc:205] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:37.251489 19073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03dd735113ab4ea6bdb696b11a4efd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.252419 15701 raft_consensus.cc:1261] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.253051 17382 raft_consensus.cc:1261] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.253213 15833 raft_consensus.cc:1261] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.253520 19073 tablet_copy_client.cc:783] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:37.253661 19073 tablet_copy_client.cc:647] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:37.253652 18653 consensus_queue.cc:1035] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.253780 18401 consensus_queue.cc:1035] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.254051 18401 consensus_queue.cc:1035] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.255846 15700 raft_consensus.cc:1261] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.256675 17167 consensus_queue.cc:1035] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.257225 19073 tablet_copy_client.cc:523] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:37.258051 15701 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.258491 19073 tablet_bootstrap.cc:492] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap starting.
I20240502 14:00:37.259033 18653 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.259577 15701 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.260389 15415 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 186a946321e445b58b0097c658847ebe with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.261615 19169 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.261906 15700 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.263231 15570 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.266320 15570 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.271855 15421 catalog_manager.cc:5484] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.272815 15422 catalog_manager.cc:5484] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 6 to 7, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.275314 17646 raft_consensus.cc:1203] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:37.275924 15700 consensus_queue.cc:237] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:37.276453 15833 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.276477 17382 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.277483 15962 raft_consensus.cc:1261] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:37.278115 18605 consensus_queue.cc:1035] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:37.278940 15570 raft_consensus.cc:1261] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:37.279330 18605 consensus_queue.cc:1035] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:37.280614 15964 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
W20240502 14:00:37.281177 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e51db70f5f284c4da3c1669fddb41b00. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.285818 17382 raft_consensus.cc:1203] T 9870bc797f184a6284ff6ca4f78b3826 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:37.288853 17231 tablet_service.cc:1514] Processing DeleteTablet for tablet 186a946321e445b58b0097c658847ebe with delete_type TABLET_DATA_TOMBSTONED (TS f688370715454d6e814a78fcf85ac5c3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60394
I20240502 14:00:37.289397 15571 consensus_queue.cc:237] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }
I20240502 14:00:37.290678 17382 raft_consensus.cc:1261] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.291078 18653 consensus_queue.cc:1035] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:37.291316 15833 raft_consensus.cc:1261] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.293095 18653 consensus_queue.cc:1035] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:37.293401 18605 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.295650 15962 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
W20240502 14:00:37.296088 15414 catalog_manager.cc:4797] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): delete failed for tablet 186a946321e445b58b0097c658847ebe because the tablet was not found. No further retry: Not found: Tablet not found: 186a946321e445b58b0097c658847ebe
I20240502 14:00:37.296353 19073 tablet_bootstrap.cc:492] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:37.296631 19073 tablet_bootstrap.cc:492] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap complete.
I20240502 14:00:37.296805 19073 ts_tablet_manager.cc:1367] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25: Time spent bootstrapping tablet: real 0.038s user 0.026s sys 0.000s
I20240502 14:00:37.297287 19073 raft_consensus.cc:348] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:37.297403 19073 raft_consensus.cc:729] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c3658c946994e2ba9f41b64aa327d25, State: Initialized, Role: LEARNER
I20240502 14:00:37.297513 19073 consensus_queue.cc:260] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:37.297817 19073 ts_tablet_manager.cc:1398] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:37.299050 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session 3c3658c946994e2ba9f41b64aa327d25-03dd735113ab4ea6bdb696b11a4efd7f received from {username='slave'} at 127.8.86.134:56803
I20240502 14:00:37.299129 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session 3c3658c946994e2ba9f41b64aa327d25-03dd735113ab4ea6bdb696b11a4efd7f on tablet 03dd735113ab4ea6bdb696b11a4efd7f with peer 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:37.299544 15426 catalog_manager.cc:5484] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.300256 18400 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.301647 15571 consensus_queue.cc:237] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }
I20240502 14:00:37.301676 15831 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.302601 17381 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.302923 17380 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.303411 15415 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 186a946321e445b58b0097c658847ebe with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.303419 18653 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:37.304803 17876 raft_consensus.cc:1050] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd: attempting to promote NON_VOTER 2254b3ee3db647598d781e467584f78d to VOTER
I20240502 14:00:37.305320 17876 consensus_queue.cc:237] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.305328 17647 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.305548 18653 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.307125 15701 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.307439 15415 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 436c127af87c49e8b7bf9b6571f7dc30 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 14:00:37.307577 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e48dda3e62e14e46b03d79648e3e2773. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.308594 15569 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.309559 18566 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.309613 19048 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:37.309935 18566 raft_consensus.cc:1050] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd: attempting to promote NON_VOTER 8fea283177394bd28d7eacf4fb744d59 to VOTER
I20240502 14:00:37.310272 18566 consensus_queue.cc:237] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:37.311784 15701 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.311939 15425 catalog_manager.cc:5484] T 186a946321e445b58b0097c658847ebe P 3c3658c946994e2ba9f41b64aa327d25 reported cstate change: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.313405 17514 raft_consensus.cc:1261] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.313818 18418 consensus_queue.cc:1035] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.313858 17380 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.315305 15681 tablet_service.cc:1514] Processing DeleteTablet for tablet 186a946321e445b58b0097c658847ebe with delete_type TABLET_DATA_TOMBSTONED (TS 8484181c4ece483c8b408817fab1a917 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:38324
I20240502 14:00:37.315755 15832 raft_consensus.cc:1261] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.316195 18418 consensus_queue.cc:1035] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.316999 15569 raft_consensus.cc:1261] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.317505 15701 consensus_queue.cc:237] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:37.317775 18851 tablet_bootstrap.cc:492] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:37.318094 18851 tablet_bootstrap.cc:492] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap complete.
I20240502 14:00:37.317973 15425 catalog_manager.cc:5484] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 7 to 8, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
W20240502 14:00:37.319936 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7981c3f46d8483caae82fbf2296e762. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.320408 15962 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.320703 15571 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.320755 15811 tablet_service.cc:1514] Processing DeleteTablet for tablet 436c127af87c49e8b7bf9b6571f7dc30 with delete_type TABLET_DATA_TOMBSTONED (TS 82d7b7201cfc431d90baca1ad0a397cf not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:54382
I20240502 14:00:37.321285 19228 tablet_replica.cc:330] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf: stopping tablet replica
I20240502 14:00:37.321321 18447 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.321672 19228 raft_consensus.cc:2229] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.321905 18447 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.322079 19228 raft_consensus.cc:2258] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.322470 17647 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.322902 18584 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.323388 19228 ts_tablet_manager.cc:1875] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.324074 18447 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:37.324205 18418 consensus_queue.cc:1035] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.327170 15420 catalog_manager.cc:5484] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.327430 19228 ts_tablet_manager.cc:1888] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:00:37.327502 19228 log.cc:1196] T 436c127af87c49e8b7bf9b6571f7dc30 P 82d7b7201cfc431d90baca1ad0a397cf: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/wals/436c127af87c49e8b7bf9b6571f7dc30
I20240502 14:00:37.327333 15962 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:37.328401 15833 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.328390 15416 catalog_manager.cc:4830] TS 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531): tablet 436c127af87c49e8b7bf9b6571f7dc30 (table rebalance_test_table_5 [id=0af7e16dce074d40ae6277d41e034322]) successfully deleted
I20240502 14:00:37.328982 18418 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.329409 15571 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:37.329571 18418 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.333024 18871 tablet_bootstrap.cc:492] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:37.332921 17647 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.333309 18871 tablet_bootstrap.cc:492] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3: Bootstrap complete.
I20240502 14:00:37.333329 15571 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.334117 15422 catalog_manager.cc:5484] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.319854 17515 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.318267 18851 ts_tablet_manager.cc:1367] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59: Time spent bootstrapping tablet: real 0.729s user 0.023s sys 0.003s
I20240502 14:00:37.339174 19222 tablet_replica.cc:330] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917: stopping tablet replica
I20240502 14:00:37.339231 15833 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.338688 18851 raft_consensus.cc:348] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.339856 18851 raft_consensus.cc:729] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8fea283177394bd28d7eacf4fb744d59, State: Initialized, Role: LEARNER
I20240502 14:00:37.339988 18851 consensus_queue.cc:260] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.342474 15962 consensus_queue.cc:237] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.343046 18917 tablet_bootstrap.cc:492] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:37.343330 18917 tablet_bootstrap.cc:492] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59: Bootstrap complete.
I20240502 14:00:37.343518 18917 ts_tablet_manager.cc:1367] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59: Time spent bootstrapping tablet: real 0.476s user 0.027s sys 0.001s
W20240502 14:00:37.344177 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e48dda3e62e14e46b03d79648e3e2773. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.344497 17647 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.343987 18917 raft_consensus.cc:348] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.344597 18917 raft_consensus.cc:729] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8fea283177394bd28d7eacf4fb744d59, State: Initialized, Role: LEARNER
I20240502 14:00:37.344718 18917 consensus_queue.cc:260] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.344857 15570 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.344967 18917 ts_tablet_manager.cc:1398] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.345616 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session 8fea283177394bd28d7eacf4fb744d59-22eef55211ca4f65979022d2f4ae829f received from {username='slave'} at 127.8.86.135:39871
I20240502 14:00:37.345607 18584 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.345690 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session 8fea283177394bd28d7eacf4fb744d59-22eef55211ca4f65979022d2f4ae829f on tablet 22eef55211ca4f65979022d2f4ae829f with peer 8fea283177394bd28d7eacf4fb744d59
I20240502 14:00:37.345880 18584 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.347224 18851 ts_tablet_manager.cc:1398] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 14:00:37.347680 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session 8fea283177394bd28d7eacf4fb744d59-7c57cab68b764db0917ed98e74757ff0 received from {username='slave'} at 127.8.86.135:39871
I20240502 14:00:37.347759 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session 8fea283177394bd28d7eacf4fb744d59-7c57cab68b764db0917ed98e74757ff0 on tablet 7c57cab68b764db0917ed98e74757ff0 with peer 8fea283177394bd28d7eacf4fb744d59
I20240502 14:00:37.347906 18566 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.348457 17647 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.349395 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e48dda3e62e14e46b03d79648e3e2773 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.349857 15569 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.350739 15422 catalog_manager.cc:5484] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 6 to 7, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.352206 15571 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.354346 15811 tablet_service.cc:1514] Processing DeleteTablet for tablet e48dda3e62e14e46b03d79648e3e2773 with delete_type TABLET_DATA_TOMBSTONED (TS 82d7b7201cfc431d90baca1ad0a397cf not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54382
I20240502 14:00:37.354691 19228 tablet_replica.cc:330] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf: stopping tablet replica
I20240502 14:00:37.354878 19228 raft_consensus.cc:2229] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.355123 19228 raft_consensus.cc:2258] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.356083 19228 ts_tablet_manager.cc:1875] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.342890 19100 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.357548 19222 raft_consensus.cc:2229] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.357811 19222 raft_consensus.cc:2258] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.358553 18871 ts_tablet_manager.cc:1367] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3: Time spent bootstrapping tablet: real 0.750s user 0.028s sys 0.000s
I20240502 14:00:37.358461 17515 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:37.359104 18871 raft_consensus.cc:348] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:37.359226 18871 raft_consensus.cc:729] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f688370715454d6e814a78fcf85ac5c3, State: Initialized, Role: LEARNER
I20240502 14:00:37.359328 18871 consensus_queue.cc:260] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:37.359642 18566 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.360086 18871 ts_tablet_manager.cc:1398] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.360977 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session f688370715454d6e814a78fcf85ac5c3-1c004727e7c64f33bc9492184b77fde9 received from {username='slave'} at 127.8.86.133:37489
I20240502 14:00:37.361060 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session f688370715454d6e814a78fcf85ac5c3-1c004727e7c64f33bc9492184b77fde9 on tablet 1c004727e7c64f33bc9492184b77fde9 with peer f688370715454d6e814a78fcf85ac5c3
I20240502 14:00:37.362267 15832 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.367331 19222 ts_tablet_manager.cc:1875] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.367419 17514 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.367548 15571 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.370347 15422 catalog_manager.cc:5484] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 4 to 5, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.371531 19222 ts_tablet_manager.cc:1888] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:00:37.373760 18897 tablet_bootstrap.cc:492] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:37.374074 18897 tablet_bootstrap.cc:492] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3: Bootstrap complete.
I20240502 14:00:37.374264 18897 ts_tablet_manager.cc:1367] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3: Time spent bootstrapping tablet: real 0.261s user 0.028s sys 0.000s
I20240502 14:00:37.374771 18897 raft_consensus.cc:348] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:37.374902 18897 raft_consensus.cc:729] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f688370715454d6e814a78fcf85ac5c3, State: Initialized, Role: LEARNER
I20240502 14:00:37.375015 18897 consensus_queue.cc:260] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:37.375227 15962 consensus_queue.cc:237] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:37.376592 17514 raft_consensus.cc:1261] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.377148 15832 raft_consensus.cc:1261] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.377596 18566 consensus_queue.cc:1035] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.377839 18418 consensus_queue.cc:1035] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.378393 18897 ts_tablet_manager.cc:1398] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:37.378799 15984 tablet_copy_service.cc:342] P 4d0f8f311e834d81bd54b9f8e7923bcd: Request end of tablet copy session f688370715454d6e814a78fcf85ac5c3-837e8b1235a44131bdaef24b763d1cac received from {username='slave'} at 127.8.86.133:49061
I20240502 14:00:37.378880 15984 tablet_copy_service.cc:433] P 4d0f8f311e834d81bd54b9f8e7923bcd: ending tablet copy session f688370715454d6e814a78fcf85ac5c3-837e8b1235a44131bdaef24b763d1cac on tablet 837e8b1235a44131bdaef24b763d1cac with peer f688370715454d6e814a78fcf85ac5c3
I20240502 14:00:37.371685 19222 log.cc:1196] T 186a946321e445b58b0097c658847ebe P 8484181c4ece483c8b408817fab1a917: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/wals/186a946321e445b58b0097c658847ebe
I20240502 14:00:37.372067 19100 raft_consensus.cc:1050] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917: attempting to promote NON_VOTER f688370715454d6e814a78fcf85ac5c3 to VOTER
I20240502 14:00:37.382078 19100 consensus_queue.cc:237] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:37.383056 18566 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.383308 19001 tablet_bootstrap.cc:492] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:37.383616 19001 tablet_bootstrap.cc:492] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3: Bootstrap complete.
I20240502 14:00:37.383827 18502 raft_consensus.cc:1050] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917: attempting to promote NON_VOTER 8fea283177394bd28d7eacf4fb744d59 to VOTER
I20240502 14:00:37.383842 19001 ts_tablet_manager.cc:1367] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3: Time spent bootstrapping tablet: real 0.494s user 0.029s sys 0.000s
I20240502 14:00:37.384183 15832 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.384297 19001 raft_consensus.cc:348] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.384456 19001 raft_consensus.cc:729] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f688370715454d6e814a78fcf85ac5c3, State: Initialized, Role: LEARNER
I20240502 14:00:37.384590 19001 consensus_queue.cc:260] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.385214 15413 catalog_manager.cc:4830] TS 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): tablet 186a946321e445b58b0097c658847ebe (table rebalance_test_table_8 [id=a148b6d170714c99a6024d0a3efa85c8]) successfully deleted
I20240502 14:00:37.385556 19001 ts_tablet_manager.cc:1398] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:37.386329 17251 raft_consensus.cc:1261] T b07f4db6f82e4a56accdbf1bbbc5f965 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.386766 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session f688370715454d6e814a78fcf85ac5c3-22eef55211ca4f65979022d2f4ae829f received from {username='slave'} at 127.8.86.133:37489
I20240502 14:00:37.386840 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session f688370715454d6e814a78fcf85ac5c3-22eef55211ca4f65979022d2f4ae829f on tablet 22eef55211ca4f65979022d2f4ae829f with peer f688370715454d6e814a78fcf85ac5c3
I20240502 14:00:37.386799 18502 consensus_queue.cc:237] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:37.388303 17251 raft_consensus.cc:1203] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:00:37.389427 17251 raft_consensus.cc:1203] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:00:37.389472 15964 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.390138 15569 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:00:37.390786 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7981c3f46d8483caae82fbf2296e762. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.390894 18174 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.391340 18174 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.391786 18174 consensus_queue.cc:1035] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.393400 17251 raft_consensus.cc:1203] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:00:37.393455 17514 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.393548 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1a5e33f337bd47b2a6b11b29a5eacad8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.395009 15425 catalog_manager.cc:5484] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 5 to 6, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.395313 19099 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.395956 17647 raft_consensus.cc:1203] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:00:37.397012 19099 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:37.397801 15549 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a5e33f337bd47b2a6b11b29a5eacad8 with delete_type TABLET_DATA_TOMBSTONED (TS 1eb0e3a6c782469fa8110f6bb6a585cf not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44898
I20240502 14:00:37.400226 15964 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:37.400221 15569 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:37.401937 15571 raft_consensus.cc:1261] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.403128 18174 consensus_queue.cc:1035] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.403308 15420 catalog_manager.cc:5484] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:37.404946 17514 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:37.406891 15962 raft_consensus.cc:1261] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.407243 18502 consensus_queue.cc:1035] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.407848 17515 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.408499 15963 consensus_queue.cc:237] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.408463 17380 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
W20240502 14:00:37.410401 15899 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e76bd0ecd3e4c168e141fb7984af3ec. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.410665 15571 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:37.411124 15701 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:37.411617 18418 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:37.411770 17876 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:37.416522 15700 consensus_queue.cc:237] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:37.419155 15569 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.420029 15963 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.420150 18447 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.420825 18584 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.422107 19228 ts_tablet_manager.cc:1888] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:00:37.422179 19228 log.cc:1196] T e48dda3e62e14e46b03d79648e3e2773 P 82d7b7201cfc431d90baca1ad0a397cf: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/wals/e48dda3e62e14e46b03d79648e3e2773
I20240502 14:00:37.422279 19100 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.423525 15416 catalog_manager.cc:4830] TS 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531): tablet e48dda3e62e14e46b03d79648e3e2773 (table rebalance_test_table_9 [id=17d53f89ea7b4f918ed8bb35d44f6401]) successfully deleted
I20240502 14:00:37.424127 19259 tablet_replica.cc:330] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf: stopping tablet replica
I20240502 14:00:37.424643 19259 raft_consensus.cc:2229] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.424916 19259 raft_consensus.cc:2258] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.424820 15701 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.425099 15427 catalog_manager.cc:5484] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.426352 18605 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.426638 17515 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.427103 19100 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.426287 19259 ts_tablet_manager.cc:1875] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.428328 15569 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.429234 15427 catalog_manager.cc:5484] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.429929 19259 ts_tablet_manager.cc:1888] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:00:37.430051 19259 log.cc:1196] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/wals/1a5e33f337bd47b2a6b11b29a5eacad8
I20240502 14:00:37.431084 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a7981c3f46d8483caae82fbf2296e762 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.431126 15963 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.431478 17515 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.432607 17362 tablet_service.cc:1514] Processing DeleteTablet for tablet a7981c3f46d8483caae82fbf2296e762 with delete_type TABLET_DATA_TOMBSTONED (TS 3c3658c946994e2ba9f41b64aa327d25 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37172
W20240502 14:00:37.433049 15413 catalog_manager.cc:4797] TS 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): delete failed for tablet a7981c3f46d8483caae82fbf2296e762 because the tablet was not found. No further retry: Not found: Tablet not found: a7981c3f46d8483caae82fbf2296e762
I20240502 14:00:37.436929 15571 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.437690 15415 catalog_manager.cc:4830] TS 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237): tablet 1a5e33f337bd47b2a6b11b29a5eacad8 (table rebalance_test_table_7 [id=fc947d3d4f264291a81e7e95cb84f48a]) successfully deleted
I20240502 14:00:37.438920 17250 raft_consensus.cc:1203] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:37.440058 18653 raft_consensus.cc:1050] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf: attempting to promote NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 to VOTER
I20240502 14:00:37.440429 18653 consensus_queue.cc:237] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.443235 15962 consensus_queue.cc:237] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:37.444423 15832 raft_consensus.cc:1261] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.444666 17380 raft_consensus.cc:1261] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.444972 19048 consensus_queue.cc:1035] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.445096 18401 consensus_queue.cc:1035] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.445569 15963 raft_consensus.cc:1261] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.445951 18401 consensus_queue.cc:1035] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.447355 15961 consensus_queue.cc:237] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:37.447576 17515 raft_consensus.cc:1261] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.447917 18401 consensus_queue.cc:1035] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:00:37.449218 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 837e8b1235a44131bdaef24b763d1cac. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.449939 17250 raft_consensus.cc:1261] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:00:37.450347 15898 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e76bd0ecd3e4c168e141fb7984af3ec. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.450559 17876 consensus_queue.cc:1035] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:00:37.450819 15899 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e76bd0ecd3e4c168e141fb7984af3ec. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.451772 15569 raft_consensus.cc:1261] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.452116 15701 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.452097 19218 consensus_queue.cc:1035] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.452430 15700 raft_consensus.cc:1261] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.452929 18566 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.453089 15571 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.453305 18566 consensus_queue.cc:1035] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.453634 18584 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.453728 18566 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.455096 18174 raft_consensus.cc:2932] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.454710 15571 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.455582 17251 raft_consensus.cc:2932] T b07f4db6f82e4a56accdbf1bbbc5f965 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.458606 15421 catalog_manager.cc:5484] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 4 to 5, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:37.459064 15701 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.459179 15571 raft_consensus.cc:2932] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.459794 15425 catalog_manager.cc:5484] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.461720 19048 raft_consensus.cc:1013] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: attempt to promote peer 8fea283177394bd28d7eacf4fb744d59: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:00:37.463601 18418 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.465512 15700 consensus_queue.cc:237] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:37.466118 18401 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.466684 17251 raft_consensus.cc:1261] T b07f4db6f82e4a56accdbf1bbbc5f965 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.467047 18502 consensus_queue.cc:1035] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.468343 17514 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.469065 15569 raft_consensus.cc:1261] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.469517 15700 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.471287 17380 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.472474 15420 catalog_manager.cc:5484] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.472882 15421 catalog_manager.cc:5484] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59 reported cstate change: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.473359 19048 raft_consensus.cc:1050] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf: attempting to promote NON_VOTER 8fea283177394bd28d7eacf4fb744d59 to VOTER
I20240502 14:00:37.473364 18506 raft_consensus.cc:1050] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf: attempting to promote NON_VOTER 2254b3ee3db647598d781e467584f78d to VOTER
I20240502 14:00:37.473742 19048 consensus_queue.cc:237] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:37.473882 18506 consensus_queue.cc:237] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:37.475370 17251 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.475781 16899 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.476527 17647 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.476723 15700 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.477156 18502 consensus_queue.cc:1035] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.477335 16899 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.479949 16899 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.479970 17380 raft_consensus.cc:1261] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.480623 17514 raft_consensus.cc:1261] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.480659 18502 raft_consensus.cc:2932] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.480834 19001 ts_tablet_manager.cc:902] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3: Initiating tablet copy from peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:37.481038 18400 consensus_queue.cc:1035] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.481326 19001 tablet_copy_client.cc:308] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.132:45719
I20240502 14:00:37.481794 15984 tablet_copy_service.cc:140] P 4d0f8f311e834d81bd54b9f8e7923bcd: Received BeginTabletCopySession request for tablet 387bdd8ae3b34f588e2170b061a5bc05 from peer f688370715454d6e814a78fcf85ac5c3 ({username='slave'} at 127.8.86.133:49061)
I20240502 14:00:37.481904 15984 tablet_copy_service.cc:161] P 4d0f8f311e834d81bd54b9f8e7923bcd: Beginning new tablet copy session on tablet 387bdd8ae3b34f588e2170b061a5bc05 from peer f688370715454d6e814a78fcf85ac5c3 at {username='slave'} at 127.8.86.133:49061: session id = f688370715454d6e814a78fcf85ac5c3-387bdd8ae3b34f588e2170b061a5bc05
I20240502 14:00:37.482992 17248 raft_consensus.cc:2932] T b07f4db6f82e4a56accdbf1bbbc5f965 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.483217 15984 tablet_copy_source_session.cc:205] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:37.483887 18400 consensus_queue.cc:1035] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.484472 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b07f4db6f82e4a56accdbf1bbbc5f965 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.485344 19001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387bdd8ae3b34f588e2170b061a5bc05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.485941 15569 raft_consensus.cc:2932] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.488075 19001 tablet_copy_client.cc:783] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:37.488049 15427 catalog_manager.cc:5484] T b07f4db6f82e4a56accdbf1bbbc5f965 P f688370715454d6e814a78fcf85ac5c3 reported cstate change: config changed from index 5 to 6, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.488209 19001 tablet_copy_client.cc:647] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:37.491281 15944 tablet_service.cc:1514] Processing DeleteTablet for tablet b07f4db6f82e4a56accdbf1bbbc5f965 with delete_type TABLET_DATA_TOMBSTONED (TS 4d0f8f311e834d81bd54b9f8e7923bcd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56876
I20240502 14:00:37.491478 19134 tablet_replica.cc:330] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd: stopping tablet replica
I20240502 14:00:37.491600 19134 raft_consensus.cc:2229] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.491705 19134 pending_rounds.cc:62] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd: Trying to abort 1 pending ops.
I20240502 14:00:37.491765 19134 pending_rounds.cc:68] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023240959515828224 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "b07f4db6f82e4a56accdbf1bbbc5f965" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } } }
I20240502 14:00:37.491894 19134 raft_consensus.cc:2894] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 14:00:37.491946 19134 raft_consensus.cc:2866] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:00:37.491986 19134 raft_consensus.cc:2258] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.492784 19134 ts_tablet_manager.cc:1875] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.493358 19001 tablet_copy_client.cc:523] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:37.494099 15964 raft_consensus.cc:1261] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.494621 19001 tablet_bootstrap.cc:492] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3: Bootstrap starting.
I20240502 14:00:37.494691 18401 consensus_queue.cc:1035] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.495050 19134 ts_tablet_manager.cc:1888] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:00:37.495118 19134 log.cc:1196] T b07f4db6f82e4a56accdbf1bbbc5f965 P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal/wals/b07f4db6f82e4a56accdbf1bbbc5f965
I20240502 14:00:37.495625 15964 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.497092 15832 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.498570 15413 catalog_manager.cc:4830] TS 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719): tablet b07f4db6f82e4a56accdbf1bbbc5f965 (table rebalance_test_table_0 [id=fcc590955678424ab1c27b0b23f75544]) successfully deleted
I20240502 14:00:37.498850 15833 raft_consensus.cc:1261] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.501227 19218 raft_consensus.cc:1050] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd: attempting to promote NON_VOTER f688370715454d6e814a78fcf85ac5c3 to VOTER
I20240502 14:00:37.501652 19218 consensus_queue.cc:237] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:37.502524 16899 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.503957 18507 raft_consensus.cc:1050] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf: attempting to promote NON_VOTER f688370715454d6e814a78fcf85ac5c3 to VOTER
I20240502 14:00:37.504320 18507 consensus_queue.cc:237] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:37.504722 15700 raft_consensus.cc:1261] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.505064 15570 raft_consensus.cc:1261] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.505194 18418 consensus_queue.cc:1035] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.505805 18401 consensus_queue.cc:1035] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.506074 17645 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.506419 15702 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.506574 18418 consensus_queue.cc:1035] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.506753 17250 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.508698 19169 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.508715 15421 catalog_manager.cc:5484] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER, now has a pending config: VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130), VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131), VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129), VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136), VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133). New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.508991 19169 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.509415 17251 raft_consensus.cc:1261] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:00:37.509791 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 837e8b1235a44131bdaef24b763d1cac. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.509685 17645 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.510957 17251 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.511610 18506 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.511871 15569 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.512419 15700 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.512662 15570 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.512992 18506 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.513871 15570 raft_consensus.cc:858] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
W20240502 14:00:37.513935 15570 raft_consensus.cc:1565] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:37.514335 15570 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.516173 18566 consensus_queue.cc:1035] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.516701 18566 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.518425 16899 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.518594 17251 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.518692 15571 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.520504 17250 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.521279 18401 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.523095 17382 raft_consensus.cc:1203] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:00:37.523425 15964 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.523661 17514 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.525153 15833 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.525976 15702 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.526767 15427 catalog_manager.cc:5484] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3 reported cstate change: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130), VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131), VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129), VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136), VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133). New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.528712 17646 raft_consensus.cc:1203] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:00:37.530644 15423 catalog_manager.cc:5484] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.530951 15423 catalog_manager.cc:5484] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 6 to 7, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.532215 15571 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.533277 17646 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.536808 15833 consensus_queue.cc:237] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:37.537262 17251 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.537438 15570 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.539228 17250 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.539625 16899 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:37.540181 17645 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.540454 15569 consensus_queue.cc:237] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:37.541010 18234 raft_consensus.cc:1050] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf: attempting to promote NON_VOTER 2254b3ee3db647598d781e467584f78d to VOTER
I20240502 14:00:37.541466 18234 consensus_queue.cc:237] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.542447 15964 raft_consensus.cc:1261] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.542799 19048 consensus_queue.cc:1035] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:37.543108 17646 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.543251 15699 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.543601 17515 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.543926 19169 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.543987 15963 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.544039 17514 raft_consensus.cc:1261] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.544206 19169 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.544451 18401 consensus_queue.cc:1035] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:37.544490 19169 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.545313 18506 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.546628 19048 raft_consensus.cc:1050] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf: attempting to promote NON_VOTER f688370715454d6e814a78fcf85ac5c3 to VOTER
I20240502 14:00:37.546991 19048 consensus_queue.cc:237] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:37.547820 17250 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.548156 18234 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.548792 17248 raft_consensus.cc:1261] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.549106 18402 consensus_queue.cc:1035] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.540858 16899 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.008s
I20240502 14:00:37.551054 17645 raft_consensus.cc:1261] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.551190 15698 raft_consensus.cc:1261] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.551563 15961 raft_consensus.cc:1261] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.551532 18402 consensus_queue.cc:1035] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.551885 18402 consensus_queue.cc:1035] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.552094 17167 consensus_queue.cc:1035] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.556123 16899 raft_consensus.cc:1013] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 8fea283177394bd28d7eacf4fb744d59: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:00:37.556246 15962 consensus_queue.cc:237] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:37.556202 19048 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.557652 15415 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 619a8ec66ba74ecea689c17b4734253a with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.557699 17251 raft_consensus.cc:1261] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.558200 18566 consensus_queue.cc:1035] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:00:37.559525 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 837e8b1235a44131bdaef24b763d1cac. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.559839 17515 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.560438 18418 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.562139 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 837e8b1235a44131bdaef24b763d1cac with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.562819 17515 raft_consensus.cc:1203] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:00:37.562894 17251 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.563818 15571 raft_consensus.cc:1261] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.564152 15962 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.565383 15422 catalog_manager.cc:5484] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 6 to 7, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.566262 15427 catalog_manager.cc:5484] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59 reported cstate change: config changed from index 7 to 8, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.568991 17382 raft_consensus.cc:1261] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.569598 18401 consensus_queue.cc:1035] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.570534 17382 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.571820 16899 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.572043 18566 consensus_queue.cc:1035] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.572021 17382 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.573282 15569 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.573688 15570 consensus_queue.cc:237] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:37.574587 15681 tablet_service.cc:1514] Processing DeleteTablet for tablet 837e8b1235a44131bdaef24b763d1cac with delete_type TABLET_DATA_TOMBSTONED (TS 8484181c4ece483c8b408817fab1a917 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38324
I20240502 14:00:37.574831 19222 tablet_replica.cc:330] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917: stopping tablet replica
I20240502 14:00:37.574972 19222 raft_consensus.cc:2229] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.575148 19222 raft_consensus.cc:2258] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.575361 15702 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.575942 19222 ts_tablet_manager.cc:1875] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.576314 15962 raft_consensus.cc:1261] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:37.576375 17646 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.576655 18401 consensus_queue.cc:1035] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:37.578104 19222 ts_tablet_manager.cc:1888] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:00:37.578181 19222 log.cc:1196] T 837e8b1235a44131bdaef24b763d1cac P 8484181c4ece483c8b408817fab1a917: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/wals/837e8b1235a44131bdaef24b763d1cac
I20240502 14:00:37.578677 15811 tablet_service.cc:1514] Processing DeleteTablet for tablet 619a8ec66ba74ecea689c17b4734253a with delete_type TABLET_DATA_TOMBSTONED (TS 82d7b7201cfc431d90baca1ad0a397cf not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:54382
I20240502 14:00:37.579031 19228 tablet_replica.cc:330] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf: stopping tablet replica
I20240502 14:00:37.579190 19228 raft_consensus.cc:2229] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.579169 19320 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.579377 19228 raft_consensus.cc:2258] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.579878 17515 raft_consensus.cc:1261] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:37.580374 15413 catalog_manager.cc:4830] TS 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): tablet 837e8b1235a44131bdaef24b763d1cac (table rebalance_test_table_6 [id=6dcd44a470ab434a988eea52c3916f26]) successfully deleted
I20240502 14:00:37.580459 19228 ts_tablet_manager.cc:1875] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.580590 19048 consensus_queue.cc:1035] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:37.580760 17251 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.581058 19048 raft_consensus.cc:1013] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: attempt to promote peer 2254b3ee3db647598d781e467584f78d: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:00:37.581580 18507 raft_consensus.cc:1013] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer f688370715454d6e814a78fcf85ac5c3: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:00:37.583201 19048 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.583688 15701 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.585017 15701 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.586795 15963 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.587355 15415 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 619a8ec66ba74ecea689c17b4734253a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.587755 17515 raft_consensus.cc:2932] T 619a8ec66ba74ecea689c17b4734253a P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.589685 15427 catalog_manager.cc:5484] T 619a8ec66ba74ecea689c17b4734253a P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 8 to 9, VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.590020 19228 ts_tablet_manager.cc:1888] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:00:37.590097 19228 log.cc:1196] T 619a8ec66ba74ecea689c17b4734253a P 82d7b7201cfc431d90baca1ad0a397cf: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/wals/619a8ec66ba74ecea689c17b4734253a
I20240502 14:00:37.592437 16899 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.594862 15429 catalog_manager.cc:5484] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.595623 17645 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.595790 17515 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.596154 15416 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 02d3da05d82340178e27218c2ce56b67 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.596302 15416 catalog_manager.cc:4830] TS 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531): tablet 619a8ec66ba74ecea689c17b4734253a (table rebalance_test_table_1 [id=a00d0c0f593149dcb05a21e3d90dda57]) successfully deleted
I20240502 14:00:37.599009 15549 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d3da05d82340178e27218c2ce56b67 with delete_type TABLET_DATA_TOMBSTONED (TS 1eb0e3a6c782469fa8110f6bb6a585cf not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:44898
I20240502 14:00:37.600584 18234 raft_consensus.cc:1050] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf: attempting to promote NON_VOTER 8fea283177394bd28d7eacf4fb744d59 to VOTER
I20240502 14:00:37.600831 19259 tablet_replica.cc:330] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf: stopping tablet replica
I20240502 14:00:37.601011 19259 raft_consensus.cc:2229] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.601059 18234 consensus_queue.cc:237] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:37.601235 19259 raft_consensus.cc:2258] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.601205 15833 consensus_queue.cc:237] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:37.602195 19259 ts_tablet_manager.cc:1875] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.602825 17249 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.602825 17250 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:37.603436 18507 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.603744 18507 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:37.603816 17515 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.604732 19259 ts_tablet_manager.cc:1888] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:00:37.604593 17250 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.604790 19259 log.cc:1196] T 02d3da05d82340178e27218c2ce56b67 P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/wals/02d3da05d82340178e27218c2ce56b67
I20240502 14:00:37.605391 19321 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:37.605574 15415 catalog_manager.cc:4830] TS 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237): tablet 02d3da05d82340178e27218c2ce56b67 (table rebalance_test_table_1 [id=a00d0c0f593149dcb05a21e3d90dda57]) successfully deleted
I20240502 14:00:37.607245 15963 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.607597 17645 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:37.607944 17647 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.607998 18507 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.608750 19169 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:37.608840 19321 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:37.609105 15420 catalog_manager.cc:5484] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3 reported cstate change: config changed from index 6 to 7, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER, now has a pending config: VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132), VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130), VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131), VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136), NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133), VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135). New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } } pending_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.609341 19169 raft_consensus.cc:1013] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer f688370715454d6e814a78fcf85ac5c3: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:00:37.609356 17362 tablet_service.cc:1514] Processing DeleteTablet for tablet 619a8ec66ba74ecea689c17b4734253a with delete_type TABLET_DATA_TOMBSTONED (TS 3c3658c946994e2ba9f41b64aa327d25 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:37172
I20240502 14:00:37.609596 19123 tablet_replica.cc:330] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25: stopping tablet replica
I20240502 14:00:37.609725 15963 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.609911 19123 raft_consensus.cc:2229] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.610175 19123 raft_consensus.cc:2258] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.613838 18234 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.614352 17647 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.614579 15963 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.613838 19320 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.615813 17645 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.617841 17251 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.619225 17248 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.620442 15702 consensus_queue.cc:237] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:37.620791 18502 raft_consensus.cc:1050] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917: attempting to promote NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 to VOTER
I20240502 14:00:37.621138 18502 consensus_queue.cc:237] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }
I20240502 14:00:37.623463 15426 catalog_manager.cc:5484] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 7 to 8, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132), VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130), VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131), VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136), NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133), VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135). New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.624133 15569 raft_consensus.cc:1261] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.624222 15416 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 02d3da05d82340178e27218c2ce56b67 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.624361 15833 raft_consensus.cc:1261] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.624133 17515 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.624683 18605 consensus_queue.cc:1035] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.625344 18605 consensus_queue.cc:1035] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.625068 19001 tablet_bootstrap.cc:492] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240502 14:00:37.626327 19282 consensus_queue.cc:1159] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3c3658c946994e2ba9f41b64aa327d25" status { last_received { term: 1 index: 8 } last_committed_idx: 8 last_received_current_leader { term: 1 index: 8 } }
I20240502 14:00:37.626693 19001 tablet_bootstrap.cc:492] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3: Bootstrap complete.
I20240502 14:00:37.626964 19001 ts_tablet_manager.cc:1367] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3: Time spent bootstrapping tablet: real 0.132s user 0.025s sys 0.005s
I20240502 14:00:37.627102 17382 raft_consensus.cc:1261] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:00:37.627486 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e51db70f5f284c4da3c1669fddb41b00. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.627580 15963 raft_consensus.cc:1261] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.627684 18605 consensus_queue.cc:1035] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.627733 19001 raft_consensus.cc:348] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:37.627880 19001 raft_consensus.cc:729] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f688370715454d6e814a78fcf85ac5c3, State: Initialized, Role: LEARNER
I20240502 14:00:37.628001 19001 consensus_queue.cc:260] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:37.628131 18174 consensus_queue.cc:1035] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.628203 19320 raft_consensus.cc:1050] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf: attempting to promote NON_VOTER f688370715454d6e814a78fcf85ac5c3 to VOTER
I20240502 14:00:37.628633 19001 ts_tablet_manager.cc:1398] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:37.628613 19320 consensus_queue.cc:237] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:37.629930 15984 tablet_copy_service.cc:342] P 4d0f8f311e834d81bd54b9f8e7923bcd: Request end of tablet copy session f688370715454d6e814a78fcf85ac5c3-387bdd8ae3b34f588e2170b061a5bc05 received from {username='slave'} at 127.8.86.133:49061
I20240502 14:00:37.630009 15984 tablet_copy_service.cc:433] P 4d0f8f311e834d81bd54b9f8e7923bcd: ending tablet copy session f688370715454d6e814a78fcf85ac5c3-387bdd8ae3b34f588e2170b061a5bc05 on tablet 387bdd8ae3b34f588e2170b061a5bc05 with peer f688370715454d6e814a78fcf85ac5c3
I20240502 14:00:37.630533 18605 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.632120 17515 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:37.632493 17248 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:37.632483 16899 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:37.632616 15425 catalog_manager.cc:5484] T 02d3da05d82340178e27218c2ce56b67 P 2254b3ee3db647598d781e467584f78d reported cstate change: config changed from index 8 to 9, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.632882 19332 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:37.634493 17645 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:37.634902 19321 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:37.635215 15426 catalog_manager.cc:5484] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.635663 15961 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:37.636302 17514 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:37.637799 18234 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:37.638211 15963 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.639549 19169 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.639847 15961 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.640446 15571 raft_consensus.cc:1261] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.641188 15681 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d3da05d82340178e27218c2ce56b67 with delete_type TABLET_DATA_TOMBSTONED (TS 8484181c4ece483c8b408817fab1a917 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:38324
I20240502 14:00:37.641527 19222 tablet_replica.cc:330] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: stopping tablet replica
I20240502 14:00:37.641590 18174 consensus_queue.cc:1035] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.641683 19222 raft_consensus.cc:2229] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.641865 19222 raft_consensus.cc:2258] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.642282 17248 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.643510 15571 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:37.643599 17515 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.645990 17645 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.646663 19222 ts_tablet_manager.cc:1875] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.648506 19123 ts_tablet_manager.cc:1875] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.648905 19222 ts_tablet_manager.cc:1888] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:00:37.648977 19222 log.cc:1196] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/wals/02d3da05d82340178e27218c2ce56b67
I20240502 14:00:37.649447 15413 catalog_manager.cc:4830] TS 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): tablet 02d3da05d82340178e27218c2ce56b67 (table rebalance_test_table_1 [id=a00d0c0f593149dcb05a21e3d90dda57]) successfully deleted
I20240502 14:00:37.649168 15423 catalog_manager.cc:5484] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 8 to 9, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.650905 19123 ts_tablet_manager.cc:1888] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:00:37.651034 19123 log.cc:1196] T 619a8ec66ba74ecea689c17b4734253a P 3c3658c946994e2ba9f41b64aa327d25: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal/wals/619a8ec66ba74ecea689c17b4734253a
I20240502 14:00:37.651695 15413 catalog_manager.cc:4830] TS 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): tablet 619a8ec66ba74ecea689c17b4734253a (table rebalance_test_table_1 [id=a00d0c0f593149dcb05a21e3d90dda57]) successfully deleted
I20240502 14:00:37.652086 18402 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
W20240502 14:00:37.653317 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e51db70f5f284c4da3c1669fddb41b00. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.654366 17645 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.655297 17251 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.657430 17167 raft_consensus.cc:1050] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf: attempting to promote NON_VOTER 2254b3ee3db647598d781e467584f78d to VOTER
I20240502 14:00:37.657763 17167 consensus_queue.cc:237] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } }
I20240502 14:00:37.657867 15700 consensus_queue.cc:237] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:37.659632 15833 raft_consensus.cc:1261] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:37.660022 18174 consensus_queue.cc:1035] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:37.660425 15961 raft_consensus.cc:1261] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:37.660517 17647 raft_consensus.cc:1261] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:00:37.660903 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6d365c10c76416a944ec8fe409e3dc5. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.661084 18400 consensus_queue.cc:1035] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.661247 18174 consensus_queue.cc:1035] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:37.661871 17251 raft_consensus.cc:1261] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.662209 15422 catalog_manager.cc:5484] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3 reported cstate change: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.662459 18401 consensus_queue.cc:1035] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.663249 18401 raft_consensus.cc:1050] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf: attempting to promote NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 to VOTER
I20240502 14:00:37.663697 18401 consensus_queue.cc:237] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }
I20240502 14:00:37.663849 15962 raft_consensus.cc:1261] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.664490 18401 consensus_queue.cc:1035] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.665429 15832 raft_consensus.cc:1261] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.666162 15964 raft_consensus.cc:1261] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.666559 18400 consensus_queue.cc:1035] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.666805 18401 consensus_queue.cc:1035] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.667897 15427 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 22eef55211ca4f65979022d2f4ae829f with cas_config_opid_index 8: aborting the task: latest config opid_index 9; task opid_index 8
I20240502 14:00:37.668385 17380 raft_consensus.cc:1261] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.668810 18401 consensus_queue.cc:1035] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.670563 15962 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.670794 18400 raft_consensus.cc:2932] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.672818 15832 consensus_queue.cc:237] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:37.673307 15427 catalog_manager.cc:5484] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 4 to 5, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.673328 15831 raft_consensus.cc:2932] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.674835 17251 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:37.675264 17513 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:37.675407 15964 raft_consensus.cc:2932] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.675520 19332 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:37.675689 19321 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:37.676520 15960 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:37.677495 19099 raft_consensus.cc:1050] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917: attempting to promote NON_VOTER 2254b3ee3db647598d781e467584f78d to VOTER
I20240502 14:00:37.677837 17645 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:37.677959 19099 consensus_queue.cc:237] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:37.678210 19321 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:37.678699 19169 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:37.680580 17381 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.681046 18174 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.681406 17647 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.681612 19321 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.681875 18174 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.683921 15571 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.684043 15416 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 22eef55211ca4f65979022d2f4ae829f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.684332 18174 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.684530 17251 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.685479 15427 catalog_manager.cc:5484] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 9 to 10, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:37.686175 17513 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.686635 15964 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.688874 18174 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:37.690152 18917 ts_tablet_manager.cc:902] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:37.690155 15831 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.690709 18917 tablet_copy_client.cc:308] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
I20240502 14:00:37.690869 18447 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.691215 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 28e371c1a999431499e09479e8f5b44b from peer 8fea283177394bd28d7eacf4fb744d59 ({username='slave'} at 127.8.86.135:39871)
I20240502 14:00:37.691305 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 28e371c1a999431499e09479e8f5b44b from peer 8fea283177394bd28d7eacf4fb744d59 at {username='slave'} at 127.8.86.135:39871: session id = 8fea283177394bd28d7eacf4fb744d59-28e371c1a999431499e09479e8f5b44b
I20240502 14:00:37.691640 15571 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:37.691881 18400 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.692075 17647 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:37.692226 15853 tablet_copy_source_session.cc:205] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:37.693130 18917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e371c1a999431499e09479e8f5b44b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.694715 15420 catalog_manager.cc:5484] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.695338 18917 tablet_copy_client.cc:783] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:37.695492 18917 tablet_copy_client.cc:647] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:37.695593 15964 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.696403 17251 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.696831 15831 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:37.697623 15426 catalog_manager.cc:5484] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 6 to 7, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:37.700069 17380 raft_consensus.cc:2932] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.700533 15700 raft_consensus.cc:1261] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.700945 15702 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:37.701119 15701 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.701606 17167 consensus_queue.cc:1035] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.701882 17647 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.703518 15702 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.704051 17645 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.705051 15702 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.707435 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:37.708057 15701 consensus_queue.cc:237] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } }
I20240502 14:00:37.711975 17515 raft_consensus.cc:2932] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.714030 15570 consensus_queue.cc:237] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }
I20240502 14:00:37.715052 15681 tablet_service.cc:1514] Processing DeleteTablet for tablet 22eef55211ca4f65979022d2f4ae829f with delete_type TABLET_DATA_TOMBSTONED (TS 8484181c4ece483c8b408817fab1a917 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:38324
I20240502 14:00:37.715155 15569 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.715551 15831 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.715850 19222 tablet_replica.cc:330] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: stopping tablet replica
I20240502 14:00:37.715955 18605 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.716014 19222 raft_consensus.cc:2229] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.716168 19222 pending_rounds.cc:62] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: Trying to abort 1 pending ops.
I20240502 14:00:37.716223 18605 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.716245 19222 pending_rounds.cc:68] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: Aborting op as it isn't in flight: id { term: 1 index: 7 } timestamp: 7023240960168706048 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "22eef55211ca4f65979022d2f4ae829f" old_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } } new_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } } }
I20240502 14:00:37.716388 19222 raft_consensus.cc:2894] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Aborting config change with OpId 1.7: Aborted: Op aborted
I20240502 14:00:37.716454 19222 raft_consensus.cc:2866] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:00:37.716498 19222 raft_consensus.cc:2258] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.716954 17380 raft_consensus.cc:1261] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.717293 17167 consensus_queue.cc:1035] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.717583 15832 consensus_queue.cc:237] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:37.718047 15571 consensus_queue.cc:237] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } }
I20240502 14:00:37.718605 17645 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.719056 18605 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.719714 15963 raft_consensus.cc:1261] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.720326 17248 raft_consensus.cc:1261] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.720892 18401 consensus_queue.cc:1035] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:37.720957 17646 raft_consensus.cc:1261] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.721181 18401 consensus_queue.cc:1035] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.721365 17513 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:00:37.721452 17647 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:00:37.721755 19169 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:00:37.722116 19169 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:00:37.722368 18653 consensus_queue.cc:1035] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:37.723109 17251 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:00:37.723488 18506 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:00:37.723501 18605 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.726239 15963 raft_consensus.cc:1261] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.726272 18401 raft_consensus.cc:2932] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.726722 15831 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.728440 15415 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b0cd21dc897b41aca9118be6df48cc0c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.728971 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 76f07464d9bd49a9b1b9d58f6a180cf8 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.729022 15422 catalog_manager.cc:5484] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 5 to 6, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.729283 17645 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.729560 18653 consensus_queue.cc:1035] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.729709 18506 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.730453 17381 raft_consensus.cc:2932] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.730554 15569 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.732162 15416 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 22eef55211ca4f65979022d2f4ae829f with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.732496 17647 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.734076 18400 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.735344 17248 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.735611 15964 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.735755 15811 tablet_service.cc:1514] Processing DeleteTablet for tablet b0cd21dc897b41aca9118be6df48cc0c with delete_type TABLET_DATA_TOMBSTONED (TS 82d7b7201cfc431d90baca1ad0a397cf not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54382
I20240502 14:00:37.736148 17250 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.737723 15415 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1f562ad00ec34a8d9184671ed55d84d1 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.738312 15423 catalog_manager.cc:5484] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.739787 15421 catalog_manager.cc:5484] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.741019 19228 tablet_replica.cc:330] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf: stopping tablet replica
I20240502 14:00:37.740916 17251 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.741151 19228 raft_consensus.cc:2229] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.741181 18917 tablet_copy_client.cc:523] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:37.741317 19228 raft_consensus.cc:2258] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.742381 19228 ts_tablet_manager.cc:1875] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.742758 18917 tablet_bootstrap.cc:492] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59: Bootstrap starting.
I20240502 14:00:37.742864 17513 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.743728 15681 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f562ad00ec34a8d9184671ed55d84d1 with delete_type TABLET_DATA_TOMBSTONED (TS 8484181c4ece483c8b408817fab1a917 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:38324
I20240502 14:00:37.744350 15701 consensus_queue.cc:237] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } }
I20240502 14:00:37.745694 17362 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f07464d9bd49a9b1b9d58f6a180cf8 with delete_type TABLET_DATA_TOMBSTONED (TS 3c3658c946994e2ba9f41b64aa327d25 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37172
I20240502 14:00:37.745779 17645 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.746912 19228 ts_tablet_manager.cc:1888] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:00:37.747035 19228 log.cc:1196] T b0cd21dc897b41aca9118be6df48cc0c P 82d7b7201cfc431d90baca1ad0a397cf: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/wals/b0cd21dc897b41aca9118be6df48cc0c
I20240502 14:00:37.747771 19123 tablet_replica.cc:330] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25: stopping tablet replica
I20240502 14:00:37.747723 15421 catalog_manager.cc:5484] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3 reported cstate change: config changed from index 10 to 11, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.747920 19123 raft_consensus.cc:2229] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 14:00:37.748011 19123 pending_rounds.cc:62] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25: Trying to abort 1 pending ops.
I20240502 14:00:37.748076 19123 pending_rounds.cc:68] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023240960727728128 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "76f07464d9bd49a9b1b9d58f6a180cf8" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } } }
I20240502 14:00:37.748246 19123 raft_consensus.cc:2894] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240502 14:00:37.748322 19123 raft_consensus.cc:2866] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:00:37.748375 19123 raft_consensus.cc:2258] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 14:00:37.749236 15416 catalog_manager.cc:4830] TS 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531): tablet b0cd21dc897b41aca9118be6df48cc0c (table rebalance_test_table_5 [id=0af7e16dce074d40ae6277d41e034322]) successfully deleted
I20240502 14:00:37.749665 19123 ts_tablet_manager.cc:1875] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.750141 17647 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.751374 15944 tablet_service.cc:1514] Processing DeleteTablet for tablet 22eef55211ca4f65979022d2f4ae829f with delete_type TABLET_DATA_TOMBSTONED (TS 4d0f8f311e834d81bd54b9f8e7923bcd not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:56876
I20240502 14:00:37.751627 19134 tablet_replica.cc:330] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd: stopping tablet replica
I20240502 14:00:37.751761 19134 raft_consensus.cc:2229] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.751952 19134 raft_consensus.cc:2258] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.752677 19134 ts_tablet_manager.cc:1875] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.754835 19134 ts_tablet_manager.cc:1888] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 14:00:37.754895 19134 log.cc:1196] T 22eef55211ca4f65979022d2f4ae829f P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal/wals/22eef55211ca4f65979022d2f4ae829f
I20240502 14:00:37.755515 15831 consensus_queue.cc:237] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:37.755852 15569 consensus_queue.cc:237] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } }
I20240502 14:00:37.755983 15413 catalog_manager.cc:4830] TS 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719): tablet 22eef55211ca4f65979022d2f4ae829f (table rebalance_test_table_6 [id=6dcd44a470ab434a988eea52c3916f26]) successfully deleted
I20240502 14:00:37.756315 18174 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:37.757438 17250 raft_consensus.cc:1261] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:37.757741 17251 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:00:37.757825 18653 consensus_queue.cc:1035] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:37.758085 19169 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:00:37.758877 17513 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:00:37.759539 19320 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:00:37.759618 17645 raft_consensus.cc:1261] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:37.759938 15571 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.760092 19320 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.761298 18174 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:37.761868 15416 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 22eef55211ca4f65979022d2f4ae829f with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.762087 18402 consensus_queue.cc:1035] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:37.762861 17248 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.763784 19100 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.767204 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 76f07464d9bd49a9b1b9d58f6a180cf8 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.767149 15423 catalog_manager.cc:5484] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 7 to 8, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.766069 15427 catalog_manager.cc:5484] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 11 to 12, VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.767246 15963 raft_consensus.cc:2932] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.769811 17512 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.769990 19123 ts_tablet_manager.cc:1888] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:00:37.770076 19123 log.cc:1196] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 3c3658c946994e2ba9f41b64aa327d25: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-5/wal/wals/76f07464d9bd49a9b1b9d58f6a180cf8
I20240502 14:00:37.770713 15413 catalog_manager.cc:4830] TS 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): tablet 76f07464d9bd49a9b1b9d58f6a180cf8 (table rebalance_test_table_7 [id=fc947d3d4f264291a81e7e95cb84f48a]) successfully deleted
I20240502 14:00:37.770820 15571 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.771629 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:37.776068 18566 raft_consensus.cc:1050] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd: attempting to promote NON_VOTER 8fea283177394bd28d7eacf4fb744d59 to VOTER
I20240502 14:00:37.776453 18566 consensus_queue.cc:237] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:37.777720 17512 raft_consensus.cc:1261] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.778133 19285 consensus_queue.cc:1035] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.778709 15831 raft_consensus.cc:1261] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.779345 15811 tablet_service.cc:1514] Processing DeleteTablet for tablet 76f07464d9bd49a9b1b9d58f6a180cf8 with delete_type TABLET_DATA_TOMBSTONED (TS 82d7b7201cfc431d90baca1ad0a397cf not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:54382
I20240502 14:00:37.779628 19285 consensus_queue.cc:1035] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.780012 18401 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.780395 19228 tablet_replica.cc:330] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: stopping tablet replica
I20240502 14:00:37.780581 19228 raft_consensus.cc:2229] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.780814 19228 raft_consensus.cc:2258] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.781471 15415 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1f562ad00ec34a8d9184671ed55d84d1 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.781594 15701 raft_consensus.cc:1261] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.782176 17248 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.782311 17644 raft_consensus.cc:2932] T 1f562ad00ec34a8d9184671ed55d84d1 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.782840 19285 consensus_queue.cc:1035] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.784844 18653 raft_consensus.cc:1050] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf: attempting to promote NON_VOTER 2254b3ee3db647598d781e467584f78d to VOTER
I20240502 14:00:37.785243 18653 consensus_queue.cc:237] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } }
I20240502 14:00:37.785539 15571 consensus_queue.cc:237] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:37.786676 15832 raft_consensus.cc:1261] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.787035 17513 raft_consensus.cc:1261] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.787199 17167 consensus_queue.cc:1035] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.787376 19321 raft_consensus.cc:1050] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf: attempting to promote NON_VOTER f688370715454d6e814a78fcf85ac5c3 to VOTER
I20240502 14:00:37.787791 19321 consensus_queue.cc:237] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:37.788079 15700 raft_consensus.cc:1261] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.788067 18400 consensus_queue.cc:1035] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.788736 17167 consensus_queue.cc:1035] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.789431 17644 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.789587 17646 raft_consensus.cc:1261] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.789805 19321 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.790262 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:37.790993 19228 ts_tablet_manager.cc:1875] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.791101 17511 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.792187 15830 raft_consensus.cc:1261] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:00:37.792866 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d81f14b7c1d1468196d0fe6d67fe93a7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:37.792897 18402 consensus_queue.cc:1035] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.793030 15963 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.793226 18402 consensus_queue.cc:1035] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.793174 18401 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.794267 15830 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.794471 19228 ts_tablet_manager.cc:1888] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:00:37.794533 19228 log.cc:1196] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/wals/76f07464d9bd49a9b1b9d58f6a180cf8
I20240502 14:00:37.795152 15416 catalog_manager.cc:4830] TS 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531): tablet 76f07464d9bd49a9b1b9d58f6a180cf8 (table rebalance_test_table_7 [id=fc947d3d4f264291a81e7e95cb84f48a]) successfully deleted
I20240502 14:00:37.795575 19321 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.796001 18605 raft_consensus.cc:2932] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.796905 15569 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.797323 19321 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.798247 15830 raft_consensus.cc:2932] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.798352 15571 raft_consensus.cc:2932] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.798645 17248 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.799031 19169 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.801226 15426 catalog_manager.cc:5484] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 4 to 5, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.804610 19222 ts_tablet_manager.cc:1875] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.806993 19222 ts_tablet_manager.cc:1888] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:00:37.807083 19222 log.cc:1196] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/wals/22eef55211ca4f65979022d2f4ae829f
I20240502 14:00:37.807567 19222 tablet_replica.cc:330] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917: stopping tablet replica
I20240502 14:00:37.807696 19222 raft_consensus.cc:2229] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.807853 19222 raft_consensus.cc:2258] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.804051 17645 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.808570 15413 catalog_manager.cc:4830] TS 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): tablet 22eef55211ca4f65979022d2f4ae829f (table rebalance_test_table_6 [id=6dcd44a470ab434a988eea52c3916f26]) successfully deleted
I20240502 14:00:37.811623 19222 ts_tablet_manager.cc:1875] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.812827 15415 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d81f14b7c1d1468196d0fe6d67fe93a7 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.813769 17381 raft_consensus.cc:1203] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:00:37.814814 19222 ts_tablet_manager.cc:1888] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:00:37.814889 19222 log.cc:1196] T 1f562ad00ec34a8d9184671ed55d84d1 P 8484181c4ece483c8b408817fab1a917: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/wals/1f562ad00ec34a8d9184671ed55d84d1
I20240502 14:00:37.815191 15421 catalog_manager.cc:5484] T 1f562ad00ec34a8d9184671ed55d84d1 P f688370715454d6e814a78fcf85ac5c3 reported cstate change: config changed from index 8 to 9, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.815419 15413 catalog_manager.cc:4830] TS 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): tablet 1f562ad00ec34a8d9184671ed55d84d1 (table rebalance_test_table_2 [id=b36f1f16cb3d4b8bae9303e2976dc2e7]) successfully deleted
I20240502 14:00:37.819348 15425 catalog_manager.cc:5484] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 6 to 7, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.819674 19394 ts_tablet_manager.cc:902] T 7c57cab68b764db0917ed98e74757ff0 P 2254b3ee3db647598d781e467584f78d: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:37.821262 15944 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f562ad00ec34a8d9184671ed55d84d1 with delete_type TABLET_DATA_TOMBSTONED (TS 4d0f8f311e834d81bd54b9f8e7923bcd not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:56876
I20240502 14:00:37.822139 17382 raft_consensus.cc:2932] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.822180 18418 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.822631 15831 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.823519 16899 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.824314 19134 tablet_replica.cc:330] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd: stopping tablet replica
I20240502 14:00:37.824534 19134 raft_consensus.cc:2229] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.824797 19134 raft_consensus.cc:2258] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.825500 17248 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.825677 15963 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.825946 19134 ts_tablet_manager.cc:1875] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.828126 19134 ts_tablet_manager.cc:1888] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:00:37.828256 19134 log.cc:1196] T 1f562ad00ec34a8d9184671ed55d84d1 P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal/wals/1f562ad00ec34a8d9184671ed55d84d1
I20240502 14:00:37.828527 15420 catalog_manager.cc:5484] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 4 to 5, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:37.829002 15413 catalog_manager.cc:4830] TS 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719): tablet 1f562ad00ec34a8d9184671ed55d84d1 (table rebalance_test_table_2 [id=b36f1f16cb3d4b8bae9303e2976dc2e7]) successfully deleted
I20240502 14:00:37.830000 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:37.830216 15681 tablet_service.cc:1514] Processing DeleteTablet for tablet d81f14b7c1d1468196d0fe6d67fe93a7 with delete_type TABLET_DATA_TOMBSTONED (TS 8484181c4ece483c8b408817fab1a917 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38324
I20240502 14:00:37.830791 19222 tablet_replica.cc:330] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917: stopping tablet replica
I20240502 14:00:37.830724 15701 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.830965 19222 raft_consensus.cc:2229] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.831146 19222 raft_consensus.cc:2258] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.831867 19222 ts_tablet_manager.cc:1875] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.836971 17514 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.837524 15422 catalog_manager.cc:5484] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.837929 15961 consensus_queue.cc:237] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:37.839421 15701 consensus_queue.cc:237] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }
I20240502 14:00:37.840443 19100 raft_consensus.cc:2932] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.840622 17511 raft_consensus.cc:1261] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.840981 18566 consensus_queue.cc:1035] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.842003 18653 raft_consensus.cc:1050] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf: attempting to promote NON_VOTER f688370715454d6e814a78fcf85ac5c3 to VOTER
I20240502 14:00:37.842242 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:37.842562 18653 consensus_queue.cc:237] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.842882 18566 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.845170 15963 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.845374 17248 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.845636 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d92e5885bf94c3e8e0e4dee40017699 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.845928 17167 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.845952 17511 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.846208 17167 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.847678 17382 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.848099 18400 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.848536 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:37.848791 15571 raft_consensus.cc:1261] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.849617 17381 raft_consensus.cc:1261] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.849674 15963 raft_consensus.cc:2932] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.850189 18174 consensus_queue.cc:1035] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.850586 15830 consensus_queue.cc:237] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:37.850811 15701 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:00:37.850888 15413 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 2d92e5885bf94c3e8e0e4dee40017699 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 5 but the committed config has opid_index of 6
I20240502 14:00:37.851251 15831 raft_consensus.cc:1261] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.851898 17251 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.852421 16899 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.852830 15420 catalog_manager.cc:5484] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59 reported cstate change: config changed from index 5 to 6, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.853369 15963 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.853590 18418 consensus_queue.cc:1035] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.853719 18507 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.856017 18400 raft_consensus.cc:1050] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf: attempting to promote NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 to VOTER
I20240502 14:00:37.856081 17511 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.856407 18400 consensus_queue.cc:237] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }
I20240502 14:00:37.857399 15831 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:37.857921 17167 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.858734 17381 raft_consensus.cc:1261] T 9870bc797f184a6284ff6ca4f78b3826 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.859222 17167 consensus_queue.cc:1035] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.859644 15832 raft_consensus.cc:1261] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.860704 17167 consensus_queue.cc:1035] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.865227 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:37.872170 15422 catalog_manager.cc:5484] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.873159 19001 ts_tablet_manager.cc:902] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3: Initiating tablet copy from peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:37.873571 19001 tablet_copy_client.cc:308] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.132:45719
I20240502 14:00:37.873872 18653 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.876084 17646 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.876113 15830 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.879508 15984 tablet_copy_service.cc:140] P 4d0f8f311e834d81bd54b9f8e7923bcd: Received BeginTabletCopySession request for tablet 6e76bd0ecd3e4c168e141fb7984af3ec from peer f688370715454d6e814a78fcf85ac5c3 ({username='slave'} at 127.8.86.133:49061)
I20240502 14:00:37.879643 15984 tablet_copy_service.cc:161] P 4d0f8f311e834d81bd54b9f8e7923bcd: Beginning new tablet copy session on tablet 6e76bd0ecd3e4c168e141fb7984af3ec from peer f688370715454d6e814a78fcf85ac5c3 at {username='slave'} at 127.8.86.133:49061: session id = f688370715454d6e814a78fcf85ac5c3-6e76bd0ecd3e4c168e141fb7984af3ec
I20240502 14:00:37.881422 18400 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.881883 15681 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d92e5885bf94c3e8e0e4dee40017699 with delete_type TABLET_DATA_TOMBSTONED (TS 8484181c4ece483c8b408817fab1a917 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38324
I20240502 14:00:37.882015 15984 tablet_copy_source_session.cc:205] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:37.882390 15702 raft_consensus.cc:1261] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:37.882658 18174 consensus_queue.cc:1035] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.883433 18653 consensus_queue.cc:1035] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:37.883715 19001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e76bd0ecd3e4c168e141fb7984af3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.883888 19380 ts_tablet_manager.cc:902] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:37.884426 19380 tablet_copy_client.cc:308] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:37.884656 15701 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.885903 19001 tablet_copy_client.cc:783] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:37.886032 19001 tablet_copy_client.cc:647] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:37.889218 15591 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet d81f14b7c1d1468196d0fe6d67fe93a7 from peer 2254b3ee3db647598d781e467584f78d ({username='slave'} at 127.8.86.136:58333)
I20240502 14:00:37.889324 15591 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet d81f14b7c1d1468196d0fe6d67fe93a7 from peer 2254b3ee3db647598d781e467584f78d at {username='slave'} at 127.8.86.136:58333: session id = 2254b3ee3db647598d781e467584f78d-d81f14b7c1d1468196d0fe6d67fe93a7
I20240502 14:00:37.890341 15591 tablet_copy_source_session.cc:205] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:37.891721 18402 raft_consensus.cc:1013] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: attempt to promote peer 3c3658c946994e2ba9f41b64aa327d25: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:00:37.892176 19380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81f14b7c1d1468196d0fe6d67fe93a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.892794 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:37.893936 17627 tablet_service.cc:1514] Processing DeleteTablet for tablet 22eef55211ca4f65979022d2f4ae829f with delete_type TABLET_DATA_TOMBSTONED (TS 2254b3ee3db647598d781e467584f78d not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:37604
I20240502 14:00:37.896811 19380 tablet_copy_client.cc:783] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:37.896966 19380 tablet_copy_client.cc:647] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:37.903851 15420 catalog_manager.cc:5484] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d reported cstate change: config changed from index 4 to 5, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.906522 15833 raft_consensus.cc:2932] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.918431 15571 consensus_queue.cc:237] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } }
I20240502 14:00:37.919768 17646 raft_consensus.cc:1261] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.919984 15833 raft_consensus.cc:1261] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.920236 19001 tablet_copy_client.cc:523] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:37.920522 18402 consensus_queue.cc:1035] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.920847 18402 consensus_queue.cc:1035] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.921785 19001 tablet_bootstrap.cc:492] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3: Bootstrap starting.
I20240502 14:00:37.922385 19048 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.923601 18447 raft_consensus.cc:2932] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.924103 15415 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bb9f100600a246708457280d91826fce with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.925302 15833 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.926980 15427 catalog_manager.cc:5484] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 5 to 6, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:37.927713 15571 raft_consensus.cc:2932] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.929489 19446 ts_tablet_manager.cc:902] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59: Initiating tablet copy from peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:37.930073 19446 tablet_copy_client.cc:308] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.132:45719
I20240502 14:00:37.930586 15984 tablet_copy_service.cc:140] P 4d0f8f311e834d81bd54b9f8e7923bcd: Received BeginTabletCopySession request for tablet 6e76bd0ecd3e4c168e141fb7984af3ec from peer 8fea283177394bd28d7eacf4fb744d59 ({username='slave'} at 127.8.86.135:44181)
I20240502 14:00:37.930712 15984 tablet_copy_service.cc:161] P 4d0f8f311e834d81bd54b9f8e7923bcd: Beginning new tablet copy session on tablet 6e76bd0ecd3e4c168e141fb7984af3ec from peer 8fea283177394bd28d7eacf4fb744d59 at {username='slave'} at 127.8.86.135:44181: session id = 8fea283177394bd28d7eacf4fb744d59-6e76bd0ecd3e4c168e141fb7984af3ec
I20240502 14:00:37.931965 15984 tablet_copy_source_session.cc:205] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:37.932432 15420 catalog_manager.cc:5484] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 5 to 6, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:37.932900 17381 raft_consensus.cc:2932] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.933117 19222 ts_tablet_manager.cc:1888] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:00:37.933198 19222 log.cc:1196] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8484181c4ece483c8b408817fab1a917: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/wals/d81f14b7c1d1468196d0fe6d67fe93a7
I20240502 14:00:37.933298 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1f80c8590e994a10a9b7c6c174e28173 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.933708 19222 tablet_replica.cc:330] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: stopping tablet replica
I20240502 14:00:37.933840 19222 raft_consensus.cc:2229] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.933982 19222 raft_consensus.cc:2258] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.934605 19446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e76bd0ecd3e4c168e141fb7984af3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.935387 19222 ts_tablet_manager.cc:1875] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.936764 19446 tablet_copy_client.cc:783] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:37.936908 19446 tablet_copy_client.cc:647] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:37.937646 19222 ts_tablet_manager.cc:1888] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:00:37.937763 19222 log.cc:1196] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/wals/2d92e5885bf94c3e8e0e4dee40017699
I20240502 14:00:37.939829 18507 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.940198 15811 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f80c8590e994a10a9b7c6c174e28173 with delete_type TABLET_DATA_TOMBSTONED (TS 82d7b7201cfc431d90baca1ad0a397cf not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54382
I20240502 14:00:37.940376 19228 tablet_replica.cc:330] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf: stopping tablet replica
I20240502 14:00:37.940512 19228 raft_consensus.cc:2229] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.940647 19228 raft_consensus.cc:2258] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.941274 15416 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2886dfbec1e141198b0cf7a435fac498 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.941411 19228 ts_tablet_manager.cc:1875] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.942528 17251 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.942873 15423 catalog_manager.cc:5484] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 6 to 7, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:37.943464 19228 ts_tablet_manager.cc:1888] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:00:37.943532 19228 log.cc:1196] T 1f80c8590e994a10a9b7c6c174e28173 P 82d7b7201cfc431d90baca1ad0a397cf: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/wals/1f80c8590e994a10a9b7c6c174e28173
I20240502 14:00:37.944360 15416 catalog_manager.cc:4830] TS 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531): tablet 1f80c8590e994a10a9b7c6c174e28173 (table rebalance_test_table_3 [id=12389884a03b40f68c532a06ec761e9c]) successfully deleted
I20240502 14:00:37.944501 15963 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.945012 17646 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:37.947019 18566 raft_consensus.cc:1050] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd: attempting to promote NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 to VOTER
I20240502 14:00:37.947427 18566 consensus_queue.cc:237] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:37.948510 15570 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:37.949066 16899 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.949586 17381 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.949856 15569 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.950009 19285 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.950264 19218 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.950917 15570 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:37.952512 15570 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.954998 17627 tablet_service.cc:1514] Processing DeleteTablet for tablet 2886dfbec1e141198b0cf7a435fac498 with delete_type TABLET_DATA_TOMBSTONED (TS 2254b3ee3db647598d781e467584f78d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37604
I20240502 14:00:37.955294 15413 catalog_manager.cc:4830] TS 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): tablet d81f14b7c1d1468196d0fe6d67fe93a7 (table rebalance_test_table_0 [id=fcc590955678424ab1c27b0b23f75544]) successfully deleted
I20240502 14:00:37.955463 15833 consensus_queue.cc:237] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:37.955538 15413 catalog_manager.cc:4830] TS 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): tablet 2d92e5885bf94c3e8e0e4dee40017699 (table rebalance_test_table_4 [id=278d2359eb234a2b8e569cd993a004a7]) successfully deleted
I20240502 14:00:37.955654 17251 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.955776 18402 raft_consensus.cc:2932] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.956228 19218 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.956753 15681 tablet_service.cc:1514] Processing DeleteTablet for tablet bb9f100600a246708457280d91826fce with delete_type TABLET_DATA_TOMBSTONED (TS 8484181c4ece483c8b408817fab1a917 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38324
I20240502 14:00:37.957907 19446 tablet_copy_client.cc:523] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:37.957989 15831 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:37.959285 17381 raft_consensus.cc:2932] T 9870bc797f184a6284ff6ca4f78b3826 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.959724 19394 tablet_copy_client.cc:308] T 7c57cab68b764db0917ed98e74757ff0 P 2254b3ee3db647598d781e467584f78d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
I20240502 14:00:37.960191 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 7c57cab68b764db0917ed98e74757ff0 from peer 2254b3ee3db647598d781e467584f78d ({username='slave'} at 127.8.86.136:53895)
I20240502 14:00:37.960276 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 7c57cab68b764db0917ed98e74757ff0 from peer 2254b3ee3db647598d781e467584f78d at {username='slave'} at 127.8.86.136:53895: session id = 2254b3ee3db647598d781e467584f78d-7c57cab68b764db0917ed98e74757ff0
I20240502 14:00:37.961110 15853 tablet_copy_source_session.cc:205] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:37.961925 19394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c57cab68b764db0917ed98e74757ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.964800 15420 catalog_manager.cc:5484] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 4 to 5, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:37.965191 15701 raft_consensus.cc:2932] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:37.970196 19446 tablet_bootstrap.cc:492] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59: Bootstrap starting.
I20240502 14:00:37.970539 18418 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:37.970860 15963 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.971091 17251 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.971483 16899 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:37.973513 16899 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.974350 19321 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:37.975265 19443 ts_tablet_manager.cc:902] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59: Initiating tablet copy from peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:37.976119 19443 tablet_copy_client.cc:308] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.132:45719
I20240502 14:00:37.976212 15963 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.976512 15416 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2886dfbec1e141198b0cf7a435fac498 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:37.976897 17251 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:37.978792 15422 catalog_manager.cc:5484] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:37.979645 19441 tablet_replica.cc:330] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d: stopping tablet replica
I20240502 14:00:37.979957 19441 raft_consensus.cc:2229] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:37.980163 19441 raft_consensus.cc:2258] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:37.980930 18653 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.983615 17381 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.985056 15702 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.986917 15984 tablet_copy_service.cc:140] P 4d0f8f311e834d81bd54b9f8e7923bcd: Received BeginTabletCopySession request for tablet e48dda3e62e14e46b03d79648e3e2773 from peer 8fea283177394bd28d7eacf4fb744d59 ({username='slave'} at 127.8.86.135:44181)
I20240502 14:00:37.987021 15984 tablet_copy_service.cc:161] P 4d0f8f311e834d81bd54b9f8e7923bcd: Beginning new tablet copy session on tablet e48dda3e62e14e46b03d79648e3e2773 from peer 8fea283177394bd28d7eacf4fb744d59 at {username='slave'} at 127.8.86.135:44181: session id = 8fea283177394bd28d7eacf4fb744d59-e48dda3e62e14e46b03d79648e3e2773
I20240502 14:00:37.987164 19441 ts_tablet_manager.cc:1875] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:37.987394 19048 raft_consensus.cc:1050] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf: attempting to promote NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 to VOTER
I20240502 14:00:37.987944 15984 tablet_copy_source_session.cc:205] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:37.987854 19048 consensus_queue.cc:237] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:37.988905 19443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48dda3e62e14e46b03d79648e3e2773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.989842 17250 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.990471 18400 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:37.991262 17250 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.992267 19380 tablet_copy_client.cc:523] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:37.993693 19380 tablet_bootstrap.cc:492] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: Bootstrap starting.
I20240502 14:00:37.994002 17381 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.994560 18400 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:37.996203 18653 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.996641 15570 consensus_queue.cc:237] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }
I20240502 14:00:37.997855 15702 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:37.998708 17382 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:37.998001 17251 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:38.000185 17380 raft_consensus.cc:1261] T 9870bc797f184a6284ff6ca4f78b3826 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:38.001062 19048 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:38.001014 17167 consensus_queue.cc:1035] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.001466 15549 tablet_service.cc:1514] Processing DeleteTablet for tablet 2886dfbec1e141198b0cf7a435fac498 with delete_type TABLET_DATA_TOMBSTONED (TS 1eb0e3a6c782469fa8110f6bb6a585cf not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:44898
I20240502 14:00:38.001658 19259 tablet_replica.cc:330] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: stopping tablet replica
I20240502 14:00:38.001782 19259 raft_consensus.cc:2229] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.001950 19259 raft_consensus.cc:2258] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.002849 15702 raft_consensus.cc:1261] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:38.002947 15420 catalog_manager.cc:5484] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 6 to 8, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:38.003656 15701 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:38.003778 17167 raft_consensus.cc:2932] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:38.005419 15415 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9870bc797f184a6284ff6ca4f78b3826 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:38.006384 17380 raft_consensus.cc:2932] T 9870bc797f184a6284ff6ca4f78b3826 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:38.007615 19048 consensus_queue.cc:1035] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.008729 18418 raft_consensus.cc:1013] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: attempt to promote peer f688370715454d6e814a78fcf85ac5c3: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:00:38.010116 19443 tablet_copy_client.cc:783] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:38.010155 15570 consensus_queue.cc:237] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:38.010565 19443 tablet_copy_client.cc:647] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:38.008803 15701 raft_consensus.cc:2932] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:38.012545 17248 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:38.012496 15702 consensus_queue.cc:237] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:38.013108 18402 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:38.014004 17515 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:38.014406 19100 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:38.016214 19259 ts_tablet_manager.cc:1875] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.016121 19100 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.016248 15421 catalog_manager.cc:5484] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 5 to 6, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:38.018169 17515 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.018422 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a7981c3f46d8483caae82fbf2296e762 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:38.018280 19259 ts_tablet_manager.cc:1888] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:00:38.018658 19259 log.cc:1196] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/wals/2886dfbec1e141198b0cf7a435fac498
I20240502 14:00:38.019166 15415 catalog_manager.cc:4830] TS 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237): tablet 2886dfbec1e141198b0cf7a435fac498 (table rebalance_test_table_5 [id=0af7e16dce074d40ae6277d41e034322]) successfully deleted
I20240502 14:00:38.021498 17380 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:38.021955 19048 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:38.023311 15811 tablet_service.cc:1514] Processing DeleteTablet for tablet 9870bc797f184a6284ff6ca4f78b3826 with delete_type TABLET_DATA_TOMBSTONED (TS 82d7b7201cfc431d90baca1ad0a397cf not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54382
I20240502 14:00:38.023524 19228 tablet_replica.cc:330] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf: stopping tablet replica
I20240502 14:00:38.023660 19228 raft_consensus.cc:2229] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.023581 19048 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.023775 19228 pending_rounds.cc:62] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf: Trying to abort 1 pending ops.
I20240502 14:00:38.023841 19228 pending_rounds.cc:68] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023240961458814976 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "9870bc797f184a6284ff6ca4f78b3826" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } } }
I20240502 14:00:38.023994 19228 raft_consensus.cc:2894] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 14:00:38.024060 19228 raft_consensus.cc:2866] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:00:38.024103 19228 raft_consensus.cc:2258] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.024994 15415 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dfdcf1124e8a474facc14c45e8963568 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:38.025976 17248 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.026266 17380 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.027789 15570 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:38.028275 18605 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:38.029781 15570 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.031773 15421 catalog_manager.cc:5484] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 8 to 9, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) evicted. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:38.032204 15420 catalog_manager.cc:5484] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 7 to 8, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:00:38.032589 19424 consensus_queue.cc:1159] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "82d7b7201cfc431d90baca1ad0a397cf" status { last_received { term: 1 index: 5 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 5 } }
I20240502 14:00:38.033393 19443 tablet_copy_client.cc:523] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:38.033736 19228 ts_tablet_manager.cc:1875] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.033823 15702 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:38.034164 19048 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:38.035449 19443 tablet_bootstrap.cc:492] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap starting.
I20240502 14:00:38.035769 19228 ts_tablet_manager.cc:1888] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:00:38.035888 19228 log.cc:1196] T 9870bc797f184a6284ff6ca4f78b3826 P 82d7b7201cfc431d90baca1ad0a397cf: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/wals/9870bc797f184a6284ff6ca4f78b3826
I20240502 14:00:38.036532 15416 catalog_manager.cc:4830] TS 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531): tablet 9870bc797f184a6284ff6ca4f78b3826 (table rebalance_test_table_9 [id=17d53f89ea7b4f918ed8bb35d44f6401]) successfully deleted
I20240502 14:00:38.037498 19285 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.039676 17248 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.039893 15832 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.041345 18584 raft_consensus.cc:1050] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd: attempting to promote NON_VOTER f688370715454d6e814a78fcf85ac5c3 to VOTER
I20240502 14:00:38.041823 18584 consensus_queue.cc:237] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:38.042608 17488 tablet_service.cc:1514] Processing DeleteTablet for tablet dfdcf1124e8a474facc14c45e8963568 with delete_type TABLET_DATA_TOMBSTONED (TS 8fea283177394bd28d7eacf4fb744d59 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59474
I20240502 14:00:38.043232 15832 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.043183 15421 catalog_manager.cc:5484] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20240502 14:00:38.043596 18566 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:38.044014 17381 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.044431 18566 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.044567 17380 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.044796 15700 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.045991 17251 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.046674 18566 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:38.047133 15570 consensus_queue.cc:237] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:38.048892 17250 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:38.049528 18402 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:38.049908 15944 tablet_service.cc:1514] Processing DeleteTablet for tablet a7981c3f46d8483caae82fbf2296e762 with delete_type TABLET_DATA_TOMBSTONED (TS 4d0f8f311e834d81bd54b9f8e7923bcd not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:56876
I20240502 14:00:38.050091 19134 tablet_replica.cc:330] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd: stopping tablet replica
I20240502 14:00:38.050210 19134 raft_consensus.cc:2229] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.050436 19134 raft_consensus.cc:2258] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.050576 18566 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.051965 17381 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:38.052980 19048 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:38.053218 19134 ts_tablet_manager.cc:1875] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.053939 16899 raft_consensus.cc:1050] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf: attempting to promote NON_VOTER 8fea283177394bd28d7eacf4fb744d59 to VOTER
I20240502 14:00:38.054605 16899 consensus_queue.cc:237] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:38.055207 19441 ts_tablet_manager.cc:1888] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240502 14:00:38.055303 19441 log.cc:1196] T 22eef55211ca4f65979022d2f4ae829f P 2254b3ee3db647598d781e467584f78d: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal/wals/22eef55211ca4f65979022d2f4ae829f
I20240502 14:00:38.055653 19134 ts_tablet_manager.cc:1888] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:00:38.055716 19134 log.cc:1196] T a7981c3f46d8483caae82fbf2296e762 P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal/wals/a7981c3f46d8483caae82fbf2296e762
I20240502 14:00:38.055884 15413 catalog_manager.cc:4830] TS 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): tablet 22eef55211ca4f65979022d2f4ae829f (table rebalance_test_table_6 [id=6dcd44a470ab434a988eea52c3916f26]) successfully deleted
I20240502 14:00:38.056510 17515 raft_consensus.cc:1261] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:38.056573 17381 raft_consensus.cc:1261] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:38.056879 15413 catalog_manager.cc:4830] TS 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719): tablet a7981c3f46d8483caae82fbf2296e762 (table rebalance_test_table_2 [id=b36f1f16cb3d4b8bae9303e2976dc2e7]) successfully deleted
I20240502 14:00:38.057000 19321 consensus_queue.cc:1035] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.057309 19321 consensus_queue.cc:1035] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.057345 17250 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.058643 19441 tablet_replica.cc:330] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d: stopping tablet replica
I20240502 14:00:38.058893 19441 raft_consensus.cc:2229] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Raft consensus shutting down.
I20240502 14:00:38.059051 19441 pending_rounds.cc:62] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d: Trying to abort 1 pending ops.
I20240502 14:00:38.059120 19441 pending_rounds.cc:68] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023240961177554944 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "2886dfbec1e141198b0cf7a435fac498" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } } }
I20240502 14:00:38.059279 19441 raft_consensus.cc:2894] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240502 14:00:38.059350 19441 raft_consensus.cc:2866] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:00:38.059403 19441 raft_consensus.cc:2258] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Raft consensus is shut down!
I20240502 14:00:38.060336 15960 raft_consensus.cc:1261] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:38.060590 17381 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.060681 19332 consensus_queue.cc:1035] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.060755 15832 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.063262 18401 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.064579 19441 ts_tablet_manager.cc:1875] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.065677 17381 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.065958 17250 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.066167 15415 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dfdcf1124e8a474facc14c45e8963568 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:38.068719 15420 catalog_manager.cc:5484] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 9 to 10, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:38.070055 19441 ts_tablet_manager.cc:1888] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:00:38.070143 15701 raft_consensus.cc:1261] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:38.070592 19332 consensus_queue.cc:1035] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.070978 15963 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:38.071162 15962 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:38.071425 15964 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:38.071907 19048 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:38.073007 15964 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:38.073588 19441 log.cc:1196] T 2886dfbec1e141198b0cf7a435fac498 P 2254b3ee3db647598d781e467584f78d: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal/wals/2886dfbec1e141198b0cf7a435fac498
I20240502 14:00:38.074119 15413 catalog_manager.cc:4830] TS 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): tablet 2886dfbec1e141198b0cf7a435fac498 (table rebalance_test_table_5 [id=0af7e16dce074d40ae6277d41e034322]) successfully deleted
I20240502 14:00:38.074601 15964 raft_consensus.cc:858] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
W20240502 14:00:38.074671 15964 raft_consensus.cc:1565] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 9' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:38.074846 19332 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 3c3658c946994e2ba9f41b64aa327d25: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:00:38.074996 15964 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:38.075573 15421 catalog_manager.cc:5484] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:38.078941 19222 tablet_replica.cc:330] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917: stopping tablet replica
I20240502 14:00:38.079142 19222 raft_consensus.cc:2229] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.079375 19222 raft_consensus.cc:2258] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.079886 19321 raft_consensus.cc:1050] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf: attempting to promote NON_VOTER 2254b3ee3db647598d781e467584f78d to VOTER
I20240502 14:00:38.080289 19321 consensus_queue.cc:237] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } }
I20240502 14:00:38.082087 17646 raft_consensus.cc:1261] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:38.082533 15570 consensus_queue.cc:237] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:38.082661 18506 consensus_queue.cc:1035] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.082875 15960 raft_consensus.cc:1261] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:38.083998 19321 consensus_queue.cc:1035] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.084658 17250 raft_consensus.cc:1261] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:38.085096 19321 consensus_queue.cc:1035] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.085389 15681 tablet_service.cc:1514] Processing DeleteTablet for tablet dfdcf1124e8a474facc14c45e8963568 with delete_type TABLET_DATA_TOMBSTONED (TS 8484181c4ece483c8b408817fab1a917 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:38324
I20240502 14:00:38.090631 15962 consensus_queue.cc:237] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:38.090834 16899 raft_consensus.cc:1013] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer f688370715454d6e814a78fcf85ac5c3: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:00:38.091734 17250 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:00:38.091989 17380 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:00:38.092073 17251 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:38.092248 19048 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:00:38.092417 17381 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:38.092494 18584 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:38.092756 18584 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:38.092893 19048 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:00:38.096832 17381 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.097388 17250 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.099498 15570 raft_consensus.cc:1261] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:38.099618 19048 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.100025 19321 consensus_queue.cc:1035] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.100036 15421 catalog_manager.cc:5484] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25 reported cstate change: config changed from index 10 to 11, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.103538 15415 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dfdcf1124e8a474facc14c45e8963568 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:38.108640 15944 tablet_service.cc:1514] Processing DeleteTablet for tablet dfdcf1124e8a474facc14c45e8963568 with delete_type TABLET_DATA_TOMBSTONED (TS 4d0f8f311e834d81bd54b9f8e7923bcd not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:56876
I20240502 14:00:38.108903 19134 tablet_replica.cc:330] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd: stopping tablet replica
I20240502 14:00:38.109072 19134 raft_consensus.cc:2229] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.109069 19394 tablet_copy_client.cc:783] T 7c57cab68b764db0917ed98e74757ff0 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:38.109254 19394 tablet_copy_client.cc:647] T 7c57cab68b764db0917ed98e74757ff0 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:38.109270 19134 raft_consensus.cc:2258] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.114929 19321 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:38.117413 17515 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:38.117413 17381 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:38.120716 15429 catalog_manager.cc:5484] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25 reported cstate change: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:38.122445 19320 raft_consensus.cc:1050] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf: attempting to promote NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 to VOTER
I20240502 14:00:38.122664 15964 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:38.122855 19320 consensus_queue.cc:237] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }
I20240502 14:00:38.125017 17381 raft_consensus.cc:1261] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.125082 17515 raft_consensus.cc:1261] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.125387 16899 consensus_queue.cc:1035] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:38.125524 19332 consensus_queue.cc:1035] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:38.126467 15964 raft_consensus.cc:1261] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.127075 19332 consensus_queue.cc:1035] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:38.127772 18566 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.128301 19482 ts_tablet_manager.cc:902] T 837e8b1235a44131bdaef24b763d1cac P 2254b3ee3db647598d781e467584f78d: Initiating tablet copy from peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:38.128612 15701 raft_consensus.cc:1261] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.129469 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 387bdd8ae3b34f588e2170b061a5bc05 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:38.130448 19332 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.130637 19169 consensus_queue.cc:1035] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.130801 17381 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.131367 19169 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:38.133149 17646 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.133281 17250 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.133455 15960 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:38.135309 17251 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.135421 17381 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:38.135425 15567 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.137482 18506 raft_consensus.cc:1050] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf: attempting to promote NON_VOTER f688370715454d6e814a78fcf85ac5c3 to VOTER
I20240502 14:00:38.137759 17515 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:38.137873 18506 consensus_queue.cc:237] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } }
I20240502 14:00:38.139274 17646 raft_consensus.cc:1261] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.139745 15960 raft_consensus.cc:1261] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.139621 15701 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:38.140180 19169 consensus_queue.cc:1035] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.140380 15567 raft_consensus.cc:1261] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.140537 19321 consensus_queue.cc:1035] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:38.141011 19169 consensus_queue.cc:1035] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:38.140934 15960 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.142122 17250 raft_consensus.cc:1261] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.144065 19169 consensus_queue.cc:1035] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:38.145640 15429 catalog_manager.cc:5484] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59 reported cstate change: config changed from index 6 to 7, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:38.146044 19222 ts_tablet_manager.cc:1875] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.147217 15425 catalog_manager.cc:5484] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d reported cstate change: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.148006 19134 ts_tablet_manager.cc:1875] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.148576 16899 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.150264 19134 ts_tablet_manager.cc:1888] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:00:38.150388 19134 log.cc:1196] T dfdcf1124e8a474facc14c45e8963568 P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal/wals/dfdcf1124e8a474facc14c45e8963568
I20240502 14:00:38.150575 17646 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.150892 15413 catalog_manager.cc:4830] TS 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719): tablet dfdcf1124e8a474facc14c45e8963568 (table rebalance_test_table_6 [id=6dcd44a470ab434a988eea52c3916f26]) successfully deleted
I20240502 14:00:38.152268 15964 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.159210 15429 catalog_manager.cc:5484] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d reported cstate change: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.162405 17251 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.167757 15420 catalog_manager.cc:5484] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 7 to 8, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:38.169368 19394 tablet_copy_client.cc:523] T 7c57cab68b764db0917ed98e74757ff0 P 2254b3ee3db647598d781e467584f78d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:38.169709 19446 tablet_bootstrap.cc:492] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:38.170055 19446 tablet_bootstrap.cc:492] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59: Bootstrap complete.
I20240502 14:00:38.170269 19446 ts_tablet_manager.cc:1367] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59: Time spent bootstrapping tablet: real 0.200s user 0.024s sys 0.003s
I20240502 14:00:38.170531 15701 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:38.170892 19446 raft_consensus.cc:348] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:38.171007 19446 raft_consensus.cc:729] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8fea283177394bd28d7eacf4fb744d59, State: Initialized, Role: LEARNER
I20240502 14:00:38.171109 19446 consensus_queue.cc:260] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:38.171209 19394 tablet_bootstrap.cc:492] T 7c57cab68b764db0917ed98e74757ff0 P 2254b3ee3db647598d781e467584f78d: Bootstrap starting.
I20240502 14:00:38.171401 19446 ts_tablet_manager.cc:1398] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:38.172108 15984 tablet_copy_service.cc:342] P 4d0f8f311e834d81bd54b9f8e7923bcd: Request end of tablet copy session 8fea283177394bd28d7eacf4fb744d59-6e76bd0ecd3e4c168e141fb7984af3ec received from {username='slave'} at 127.8.86.135:44181
I20240502 14:00:38.172194 15984 tablet_copy_service.cc:433] P 4d0f8f311e834d81bd54b9f8e7923bcd: ending tablet copy session 8fea283177394bd28d7eacf4fb744d59-6e76bd0ecd3e4c168e141fb7984af3ec on tablet 6e76bd0ecd3e4c168e141fb7984af3ec with peer 8fea283177394bd28d7eacf4fb744d59
I20240502 14:00:38.177492 17511 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:38.177587 19443 tablet_bootstrap.cc:492] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:38.177884 19443 tablet_bootstrap.cc:492] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap complete.
I20240502 14:00:38.178052 19443 ts_tablet_manager.cc:1367] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59: Time spent bootstrapping tablet: real 0.143s user 0.029s sys 0.000s
I20240502 14:00:38.178264 15571 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.178527 19443 raft_consensus.cc:348] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:38.178645 19443 raft_consensus.cc:729] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8fea283177394bd28d7eacf4fb744d59, State: Initialized, Role: LEARNER
I20240502 14:00:38.178753 19443 consensus_queue.cc:260] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:38.179141 19443 ts_tablet_manager.cc:1398] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:38.179615 15984 tablet_copy_service.cc:342] P 4d0f8f311e834d81bd54b9f8e7923bcd: Request end of tablet copy session 8fea283177394bd28d7eacf4fb744d59-e48dda3e62e14e46b03d79648e3e2773 received from {username='slave'} at 127.8.86.135:44181
I20240502 14:00:38.179702 15984 tablet_copy_service.cc:433] P 4d0f8f311e834d81bd54b9f8e7923bcd: ending tablet copy session 8fea283177394bd28d7eacf4fb744d59-e48dda3e62e14e46b03d79648e3e2773 on tablet e48dda3e62e14e46b03d79648e3e2773 with peer 8fea283177394bd28d7eacf4fb744d59
I20240502 14:00:38.182538 15568 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:38.183022 18566 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.183979 15568 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.185722 15568 raft_consensus.cc:858] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
W20240502 14:00:38.185783 15568 raft_consensus.cc:1565] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 8' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:38.186043 15568 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.188473 19222 ts_tablet_manager.cc:1888] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:00:38.188570 19222 log.cc:1196] T bb9f100600a246708457280d91826fce P 8484181c4ece483c8b408817fab1a917: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/wals/bb9f100600a246708457280d91826fce
I20240502 14:00:38.189100 19222 tablet_replica.cc:330] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: stopping tablet replica
I20240502 14:00:38.189246 19222 raft_consensus.cc:2229] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.189481 19222 raft_consensus.cc:2258] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.190133 15413 catalog_manager.cc:4830] TS 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): tablet bb9f100600a246708457280d91826fce (table rebalance_test_table_4 [id=278d2359eb234a2b8e569cd993a004a7]) successfully deleted
I20240502 14:00:38.200199 19465 ts_tablet_manager.cc:902] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d: Initiating tablet copy from peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:38.201644 19465 tablet_copy_client.cc:308] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:42261
I20240502 14:00:38.202168 15721 tablet_copy_service.cc:140] P 8484181c4ece483c8b408817fab1a917: Received BeginTabletCopySession request for tablet e51db70f5f284c4da3c1669fddb41b00 from peer 2254b3ee3db647598d781e467584f78d ({username='slave'} at 127.8.86.136:42777)
I20240502 14:00:38.202275 15721 tablet_copy_service.cc:161] P 8484181c4ece483c8b408817fab1a917: Beginning new tablet copy session on tablet e51db70f5f284c4da3c1669fddb41b00 from peer 2254b3ee3db647598d781e467584f78d at {username='slave'} at 127.8.86.136:42777: session id = 2254b3ee3db647598d781e467584f78d-e51db70f5f284c4da3c1669fddb41b00
I20240502 14:00:38.198712 15567 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.203437 15721 tablet_copy_source_session.cc:205] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:38.204564 19465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e51db70f5f284c4da3c1669fddb41b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.212838 19477 tablet_replica.cc:330] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59: stopping tablet replica
I20240502 14:00:38.213303 19477 raft_consensus.cc:2229] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 14:00:38.213632 19477 pending_rounds.cc:62] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59: Trying to abort 1 pending ops.
I20240502 14:00:38.214588 19477 pending_rounds.cc:68] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59: Aborting op as it isn't in flight: id { term: 1 index: 7 } timestamp: 7023240961401597952 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "dfdcf1124e8a474facc14c45e8963568" old_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } } new_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } } }
I20240502 14:00:38.214938 19477 raft_consensus.cc:2894] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Aborting config change with OpId 1.7: Aborted: Op aborted
I20240502 14:00:38.215060 19477 raft_consensus.cc:2866] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:00:38.215143 19477 raft_consensus.cc:2258] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 14:00:38.217298 15568 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.217697 19465 tablet_copy_client.cc:783] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:38.218739 19465 tablet_copy_client.cc:647] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:38.219599 15811 tablet_service.cc:1514] Processing DeleteTablet for tablet 387bdd8ae3b34f588e2170b061a5bc05 with delete_type TABLET_DATA_TOMBSTONED (TS 82d7b7201cfc431d90baca1ad0a397cf not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:54382
I20240502 14:00:38.219988 19228 tablet_replica.cc:330] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf: stopping tablet replica
I20240502 14:00:38.220157 19228 raft_consensus.cc:2229] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.220342 19228 raft_consensus.cc:2258] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.221179 19228 ts_tablet_manager.cc:1875] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.222191 19482 tablet_copy_client.cc:308] T 837e8b1235a44131bdaef24b763d1cac P 2254b3ee3db647598d781e467584f78d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.132:45719
I20240502 14:00:38.221115 15832 consensus_queue.cc:237] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }
I20240502 14:00:38.222683 15984 tablet_copy_service.cc:140] P 4d0f8f311e834d81bd54b9f8e7923bcd: Received BeginTabletCopySession request for tablet 837e8b1235a44131bdaef24b763d1cac from peer 2254b3ee3db647598d781e467584f78d ({username='slave'} at 127.8.86.136:55499)
I20240502 14:00:38.222783 15984 tablet_copy_service.cc:161] P 4d0f8f311e834d81bd54b9f8e7923bcd: Beginning new tablet copy session on tablet 837e8b1235a44131bdaef24b763d1cac from peer 2254b3ee3db647598d781e467584f78d at {username='slave'} at 127.8.86.136:55499: session id = 2254b3ee3db647598d781e467584f78d-837e8b1235a44131bdaef24b763d1cac
I20240502 14:00:38.223671 15984 tablet_copy_source_session.cc:205] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:38.224320 17381 raft_consensus.cc:1261] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:38.224504 17511 raft_consensus.cc:1261] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:38.224889 18507 consensus_queue.cc:1035] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:38.225275 19228 ts_tablet_manager.cc:1888] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:00:38.225335 19228 log.cc:1196] T 387bdd8ae3b34f588e2170b061a5bc05 P 82d7b7201cfc431d90baca1ad0a397cf: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/wals/387bdd8ae3b34f588e2170b061a5bc05
I20240502 14:00:38.226027 19169 consensus_queue.cc:1035] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:38.226812 19477 ts_tablet_manager.cc:1875] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.227339 19482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837e8b1235a44131bdaef24b763d1cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.227892 19169 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:38.228317 17381 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:38.228636 15964 raft_consensus.cc:1261] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:38.229040 18507 consensus_queue.cc:1035] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:38.230530 17511 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:38.231177 15964 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:38.232967 15429 catalog_manager.cc:5484] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25 reported cstate change: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:38.234390 15832 consensus_queue.cc:237] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } }
I20240502 14:00:38.235785 17250 raft_consensus.cc:1261] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:38.235963 17646 raft_consensus.cc:1261] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:38.236214 19321 consensus_queue.cc:1035] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:38.236515 19321 consensus_queue.cc:1035] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:38.236668 18917 tablet_bootstrap.cc:492] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:38.236706 15960 raft_consensus.cc:1261] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:38.238410 19321 consensus_queue.cc:1035] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:38.238718 18917 tablet_bootstrap.cc:492] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59: Bootstrap complete.
I20240502 14:00:38.238907 18917 ts_tablet_manager.cc:1367] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59: Time spent bootstrapping tablet: real 0.496s user 0.029s sys 0.000s
I20240502 14:00:38.240152 18917 raft_consensus.cc:348] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:38.240633 18507 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.242341 18917 raft_consensus.cc:729] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8fea283177394bd28d7eacf4fb744d59, State: Initialized, Role: LEARNER
I20240502 14:00:38.242516 18917 consensus_queue.cc:260] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:38.242800 15681 tablet_service.cc:1514] Processing DeleteTablet for tablet 03dd735113ab4ea6bdb696b11a4efd7f with delete_type TABLET_DATA_TOMBSTONED (TS 8484181c4ece483c8b408817fab1a917 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:38324
I20240502 14:00:38.242661 17646 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.243289 18917 ts_tablet_manager.cc:1398] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:38.243909 19001 tablet_bootstrap.cc:492] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:38.244995 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session 8fea283177394bd28d7eacf4fb744d59-28e371c1a999431499e09479e8f5b44b received from {username='slave'} at 127.8.86.135:39871
I20240502 14:00:38.245074 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session 8fea283177394bd28d7eacf4fb744d59-28e371c1a999431499e09479e8f5b44b on tablet 28e371c1a999431499e09479e8f5b44b with peer 8fea283177394bd28d7eacf4fb744d59
I20240502 14:00:38.245425 19001 tablet_bootstrap.cc:492] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3: Bootstrap complete.
I20240502 14:00:38.246428 19001 ts_tablet_manager.cc:1367] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3: Time spent bootstrapping tablet: real 0.325s user 0.027s sys 0.000s
I20240502 14:00:38.246264 17251 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.246973 19001 raft_consensus.cc:348] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:38.247087 19001 raft_consensus.cc:729] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f688370715454d6e814a78fcf85ac5c3, State: Initialized, Role: LEARNER
I20240502 14:00:38.247196 19001 consensus_queue.cc:260] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:38.249082 15960 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.249500 19001 ts_tablet_manager.cc:1398] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:38.253770 15420 catalog_manager.cc:5484] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.254374 15984 tablet_copy_service.cc:342] P 4d0f8f311e834d81bd54b9f8e7923bcd: Request end of tablet copy session f688370715454d6e814a78fcf85ac5c3-6e76bd0ecd3e4c168e141fb7984af3ec received from {username='slave'} at 127.8.86.133:49061
I20240502 14:00:38.254460 15984 tablet_copy_service.cc:433] P 4d0f8f311e834d81bd54b9f8e7923bcd: ending tablet copy session f688370715454d6e814a78fcf85ac5c3-6e76bd0ecd3e4c168e141fb7984af3ec on tablet 6e76bd0ecd3e4c168e141fb7984af3ec with peer f688370715454d6e814a78fcf85ac5c3
I20240502 14:00:38.256402 19477 ts_tablet_manager.cc:1888] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:00:38.256508 19477 log.cc:1196] T dfdcf1124e8a474facc14c45e8963568 P 8fea283177394bd28d7eacf4fb744d59: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-6/wal/wals/dfdcf1124e8a474facc14c45e8963568
I20240502 14:00:38.256578 19001 ts_tablet_manager.cc:902] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3: Initiating tablet copy from peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:38.256820 19482 tablet_copy_client.cc:783] T 837e8b1235a44131bdaef24b763d1cac P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:38.257100 19482 tablet_copy_client.cc:647] T 837e8b1235a44131bdaef24b763d1cac P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:38.257059 15415 catalog_manager.cc:4830] TS 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): tablet dfdcf1124e8a474facc14c45e8963568 (table rebalance_test_table_6 [id=6dcd44a470ab434a988eea52c3916f26]) successfully deleted
I20240502 14:00:38.257409 19001 tablet_copy_client.cc:308] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:42261
I20240502 14:00:38.257843 15721 tablet_copy_service.cc:140] P 8484181c4ece483c8b408817fab1a917: Received BeginTabletCopySession request for tablet e51db70f5f284c4da3c1669fddb41b00 from peer f688370715454d6e814a78fcf85ac5c3 ({username='slave'} at 127.8.86.133:54505)
I20240502 14:00:38.257944 15721 tablet_copy_service.cc:161] P 8484181c4ece483c8b408817fab1a917: Beginning new tablet copy session on tablet e51db70f5f284c4da3c1669fddb41b00 from peer f688370715454d6e814a78fcf85ac5c3 at {username='slave'} at 127.8.86.133:54505: session id = f688370715454d6e814a78fcf85ac5c3-e51db70f5f284c4da3c1669fddb41b00
I20240502 14:00:38.258880 15721 tablet_copy_source_session.cc:205] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:38.259394 15416 catalog_manager.cc:4830] TS 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531): tablet 387bdd8ae3b34f588e2170b061a5bc05 (table rebalance_test_table_1 [id=a00d0c0f593149dcb05a21e3d90dda57]) successfully deleted
I20240502 14:00:38.259534 15416 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 03dd735113ab4ea6bdb696b11a4efd7f with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:38.259660 15416 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c004727e7c64f33bc9492184b77fde9 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:38.259876 19001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e51db70f5f284c4da3c1669fddb41b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.263602 15549 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c004727e7c64f33bc9492184b77fde9 with delete_type TABLET_DATA_TOMBSTONED (TS 1eb0e3a6c782469fa8110f6bb6a585cf not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:44898
I20240502 14:00:38.263811 19259 tablet_replica.cc:330] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf: stopping tablet replica
I20240502 14:00:38.263937 19259 raft_consensus.cc:2229] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.264150 19259 raft_consensus.cc:2258] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.266731 19259 ts_tablet_manager.cc:1875] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.268905 19259 ts_tablet_manager.cc:1888] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:00:38.268968 19259 log.cc:1196] T 1c004727e7c64f33bc9492184b77fde9 P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/wals/1c004727e7c64f33bc9492184b77fde9
I20240502 14:00:38.269757 15415 catalog_manager.cc:4830] TS 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237): tablet 1c004727e7c64f33bc9492184b77fde9 (table rebalance_test_table_9 [id=17d53f89ea7b4f918ed8bb35d44f6401]) successfully deleted
I20240502 14:00:38.271354 19465 tablet_copy_client.cc:523] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:38.272810 19465 tablet_bootstrap.cc:492] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d: Bootstrap starting.
I20240502 14:00:38.273211 19380 tablet_bootstrap.cc:492] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:38.273486 19380 tablet_bootstrap.cc:492] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: Bootstrap complete.
I20240502 14:00:38.273633 19380 ts_tablet_manager.cc:1367] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: Time spent bootstrapping tablet: real 0.280s user 0.028s sys 0.000s
I20240502 14:00:38.275313 19001 tablet_copy_client.cc:783] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:38.275508 19001 tablet_copy_client.cc:647] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:38.277837 17251 raft_consensus.cc:1203] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:00:38.278302 19380 raft_consensus.cc:348] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:38.278427 19380 raft_consensus.cc:729] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2254b3ee3db647598d781e467584f78d, State: Initialized, Role: LEARNER
I20240502 14:00:38.278537 19380 consensus_queue.cc:260] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:38.278924 19380 ts_tablet_manager.cc:1398] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:00:38.279448 15591 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session 2254b3ee3db647598d781e467584f78d-d81f14b7c1d1468196d0fe6d67fe93a7 received from {username='slave'} at 127.8.86.136:58333
I20240502 14:00:38.279529 15591 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session 2254b3ee3db647598d781e467584f78d-d81f14b7c1d1468196d0fe6d67fe93a7 on tablet d81f14b7c1d1468196d0fe6d67fe93a7 with peer 2254b3ee3db647598d781e467584f78d
I20240502 14:00:38.280531 19380 ts_tablet_manager.cc:902] T b6d365c10c76416a944ec8fe409e3dc5 P 2254b3ee3db647598d781e467584f78d: Initiating tablet copy from peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:38.280944 19380 tablet_copy_client.cc:308] T b6d365c10c76416a944ec8fe409e3dc5 P 2254b3ee3db647598d781e467584f78d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:42261
I20240502 14:00:38.281411 15721 tablet_copy_service.cc:140] P 8484181c4ece483c8b408817fab1a917: Received BeginTabletCopySession request for tablet b6d365c10c76416a944ec8fe409e3dc5 from peer 2254b3ee3db647598d781e467584f78d ({username='slave'} at 127.8.86.136:42777)
I20240502 14:00:38.281514 15721 tablet_copy_service.cc:161] P 8484181c4ece483c8b408817fab1a917: Beginning new tablet copy session on tablet b6d365c10c76416a944ec8fe409e3dc5 from peer 2254b3ee3db647598d781e467584f78d at {username='slave'} at 127.8.86.136:42777: session id = 2254b3ee3db647598d781e467584f78d-b6d365c10c76416a944ec8fe409e3dc5
I20240502 14:00:38.282527 15721 tablet_copy_source_session.cc:205] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:38.283434 19380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d365c10c76416a944ec8fe409e3dc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.285861 19380 tablet_copy_client.cc:783] T b6d365c10c76416a944ec8fe409e3dc5 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:38.286054 19380 tablet_copy_client.cc:647] T b6d365c10c76416a944ec8fe409e3dc5 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:38.287854 15832 consensus_queue.cc:237] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }
I20240502 14:00:38.288903 19222 ts_tablet_manager.cc:1875] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.290620 17381 raft_consensus.cc:1261] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:38.291049 19169 consensus_queue.cc:1035] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:38.291322 19222 ts_tablet_manager.cc:1888] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 14:00:38.291451 19222 log.cc:1196] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/wals/dfdcf1124e8a474facc14c45e8963568
I20240502 14:00:38.292093 15413 catalog_manager.cc:4830] TS 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): tablet dfdcf1124e8a474facc14c45e8963568 (table rebalance_test_table_6 [id=6dcd44a470ab434a988eea52c3916f26]) successfully deleted
I20240502 14:00:38.292371 19482 tablet_copy_client.cc:523] T 837e8b1235a44131bdaef24b763d1cac P 2254b3ee3db647598d781e467584f78d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:38.292930 19169 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:38.293653 17381 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:38.293913 17511 raft_consensus.cc:1261] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:38.294529 19321 consensus_queue.cc:1035] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:38.296034 17511 raft_consensus.cc:2932] T 03dd735113ab4ea6bdb696b11a4efd7f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:38.296845 15421 catalog_manager.cc:5484] T 03dd735113ab4ea6bdb696b11a4efd7f P 3c3658c946994e2ba9f41b64aa327d25 reported cstate change: config changed from index 8 to 9, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:38.299806 15944 tablet_service.cc:1514] Processing DeleteTablet for tablet 03dd735113ab4ea6bdb696b11a4efd7f with delete_type TABLET_DATA_TOMBSTONED (TS 4d0f8f311e834d81bd54b9f8e7923bcd not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:56876
I20240502 14:00:38.300035 19134 tablet_replica.cc:330] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd: stopping tablet replica
I20240502 14:00:38.300180 19134 raft_consensus.cc:2229] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.300367 19134 raft_consensus.cc:2258] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.301342 19134 ts_tablet_manager.cc:1875] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.303516 19134 ts_tablet_manager.cc:1888] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:00:38.304670 15416 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 03dd735113ab4ea6bdb696b11a4efd7f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:38.303597 19134 log.cc:1196] T 03dd735113ab4ea6bdb696b11a4efd7f P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal/wals/03dd735113ab4ea6bdb696b11a4efd7f
I20240502 14:00:38.305545 15413 catalog_manager.cc:4830] TS 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719): tablet 03dd735113ab4ea6bdb696b11a4efd7f (table rebalance_test_table_7 [id=fc947d3d4f264291a81e7e95cb84f48a]) successfully deleted
I20240502 14:00:38.305733 19380 tablet_copy_client.cc:523] T b6d365c10c76416a944ec8fe409e3dc5 P 2254b3ee3db647598d781e467584f78d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:38.307061 19380 tablet_bootstrap.cc:492] T b6d365c10c76416a944ec8fe409e3dc5 P 2254b3ee3db647598d781e467584f78d: Bootstrap starting.
I20240502 14:00:38.307770 19001 tablet_copy_client.cc:523] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:38.309135 19001 tablet_bootstrap.cc:492] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3: Bootstrap starting.
I20240502 14:00:38.311046 17511 raft_consensus.cc:1203] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:38.321856 19482 tablet_bootstrap.cc:492] T 837e8b1235a44131bdaef24b763d1cac P 2254b3ee3db647598d781e467584f78d: Bootstrap starting.
I20240502 14:00:38.323376 18234 raft_consensus.cc:1050] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf: attempting to promote NON_VOTER 8fea283177394bd28d7eacf4fb744d59 to VOTER
I20240502 14:00:38.323967 18234 consensus_queue.cc:237] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:38.326999 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:38.326995 15701 raft_consensus.cc:1261] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:38.327481 15964 raft_consensus.cc:1261] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:00:38.328994 15766 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:38.329093 19321 consensus_queue.cc:1035] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.329329 19321 consensus_queue.cc:1035] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.329833 17511 raft_consensus.cc:1261] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:38.330193 19169 consensus_queue.cc:1035] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.332804 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:38.336043 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:38.341267 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:38.343535 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:38.356977 19222 tablet_replica.cc:330] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917: stopping tablet replica
I20240502 14:00:38.357306 19222 raft_consensus.cc:2229] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.357575 19222 raft_consensus.cc:2258] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.357594 15832 consensus_queue.cc:237] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } }
I20240502 14:00:38.359386 17646 raft_consensus.cc:1261] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:38.359856 19321 raft_consensus.cc:2932] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:38.362097 17511 raft_consensus.cc:2932] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:38.364698 19332 consensus_queue.cc:1035] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:38.365526 15964 raft_consensus.cc:2932] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:38.368379 17251 raft_consensus.cc:1261] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:38.368953 19169 consensus_queue.cc:1035] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:38.370973 16899 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.371418 17251 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.373836 17646 raft_consensus.cc:2932] T 1c004727e7c64f33bc9492184b77fde9 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.373983 15424 catalog_manager.cc:5484] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:38.374207 15429 catalog_manager.cc:5484] T 1c004727e7c64f33bc9492184b77fde9 P f688370715454d6e814a78fcf85ac5c3 reported cstate change: config changed from index 8 to 9, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:38.377820 15944 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c004727e7c64f33bc9492184b77fde9 with delete_type TABLET_DATA_TOMBSTONED (TS 4d0f8f311e834d81bd54b9f8e7923bcd not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:56876
I20240502 14:00:38.378049 19134 tablet_replica.cc:330] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd: stopping tablet replica
I20240502 14:00:38.378192 19134 raft_consensus.cc:2229] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.378413 19134 raft_consensus.cc:2258] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.379333 19134 ts_tablet_manager.cc:1875] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.381467 19134 ts_tablet_manager.cc:1888] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:00:38.381529 19134 log.cc:1196] T 1c004727e7c64f33bc9492184b77fde9 P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal/wals/1c004727e7c64f33bc9492184b77fde9
I20240502 14:00:38.381527 19222 ts_tablet_manager.cc:1875] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.382915 15413 catalog_manager.cc:4830] TS 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719): tablet 1c004727e7c64f33bc9492184b77fde9 (table rebalance_test_table_9 [id=17d53f89ea7b4f918ed8bb35d44f6401]) successfully deleted
I20240502 14:00:38.385826 19535 tablet_replica.cc:330] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: stopping tablet replica
I20240502 14:00:38.386026 19535 raft_consensus.cc:2229] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Raft consensus shutting down.
I20240502 14:00:38.386197 19535 raft_consensus.cc:2258] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Raft consensus is shut down!
I20240502 14:00:38.386982 19535 ts_tablet_manager.cc:1875] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.388653 19394 tablet_bootstrap.cc:492] T 7c57cab68b764db0917ed98e74757ff0 P 2254b3ee3db647598d781e467584f78d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:38.388918 19394 tablet_bootstrap.cc:492] T 7c57cab68b764db0917ed98e74757ff0 P 2254b3ee3db647598d781e467584f78d: Bootstrap complete.
I20240502 14:00:38.389084 19394 ts_tablet_manager.cc:1367] T 7c57cab68b764db0917ed98e74757ff0 P 2254b3ee3db647598d781e467584f78d: Time spent bootstrapping tablet: real 0.218s user 0.027s sys 0.000s
I20240502 14:00:38.390026 17511 raft_consensus.cc:1203] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:00:38.390542 19535 ts_tablet_manager.cc:1888] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:00:38.390672 19535 log.cc:1196] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal/wals/d81f14b7c1d1468196d0fe6d67fe93a7
I20240502 14:00:38.391016 19535 ts_tablet_manager.cc:902] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:38.391490 19535 tablet_copy_client.cc:308] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:38.391992 15591 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet d81f14b7c1d1468196d0fe6d67fe93a7 from peer 2254b3ee3db647598d781e467584f78d ({username='slave'} at 127.8.86.136:58333)
I20240502 14:00:38.392093 15591 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet d81f14b7c1d1468196d0fe6d67fe93a7 from peer 2254b3ee3db647598d781e467584f78d at {username='slave'} at 127.8.86.136:58333: session id = 2254b3ee3db647598d781e467584f78d-d81f14b7c1d1468196d0fe6d67fe93a7
I20240502 14:00:38.392971 15591 tablet_copy_source_session.cc:205] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:38.393728 19535 ts_tablet_manager.cc:1875] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:00:38.396699 19535 ts_tablet_manager.cc:1888] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240502 14:00:38.396919 19535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81f14b7c1d1468196d0fe6d67fe93a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.398264 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:38.398113 19535 tablet_copy_client.cc:783] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:38.398599 19535 tablet_copy_client.cc:647] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:38.399662 19535 tablet_copy_client.cc:523] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:38.401675 19535 tablet_bootstrap.cc:492] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: Bootstrap starting.
I20240502 14:00:38.404230 15701 raft_consensus.cc:2932] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:38.405241 19394 raft_consensus.cc:348] T 7c57cab68b764db0917ed98e74757ff0 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:38.405373 19394 raft_consensus.cc:729] T 7c57cab68b764db0917ed98e74757ff0 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2254b3ee3db647598d781e467584f78d, State: Initialized, Role: LEARNER
I20240502 14:00:38.405498 19394 consensus_queue.cc:260] T 7c57cab68b764db0917ed98e74757ff0 P 2254b3ee3db647598d781e467584f78d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:38.405790 19394 ts_tablet_manager.cc:1398] T 7c57cab68b764db0917ed98e74757ff0 P 2254b3ee3db647598d781e467584f78d: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20240502 14:00:38.406400 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session 2254b3ee3db647598d781e467584f78d-7c57cab68b764db0917ed98e74757ff0 received from {username='slave'} at 127.8.86.136:53895
I20240502 14:00:38.406473 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session 2254b3ee3db647598d781e467584f78d-7c57cab68b764db0917ed98e74757ff0 on tablet 7c57cab68b764db0917ed98e74757ff0 with peer 2254b3ee3db647598d781e467584f78d
I20240502 14:00:38.431875 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:38.432128 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:38.448218 19465 tablet_bootstrap.cc:492] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:38.448678 19465 tablet_bootstrap.cc:492] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d: Bootstrap complete.
I20240502 14:00:38.448925 19465 ts_tablet_manager.cc:1367] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d: Time spent bootstrapping tablet: real 0.176s user 0.027s sys 0.000s
I20240502 14:00:38.449431 19465 raft_consensus.cc:348] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:38.449667 19465 raft_consensus.cc:729] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2254b3ee3db647598d781e467584f78d, State: Initialized, Role: LEARNER
I20240502 14:00:38.449736 19001 tablet_bootstrap.cc:492] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:38.449823 19465 consensus_queue.cc:260] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:38.450098 19001 tablet_bootstrap.cc:492] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3: Bootstrap complete.
I20240502 14:00:38.450330 19001 ts_tablet_manager.cc:1367] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3: Time spent bootstrapping tablet: real 0.141s user 0.025s sys 0.000s
I20240502 14:00:38.450330 19465 ts_tablet_manager.cc:1398] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:38.450785 15721 tablet_copy_service.cc:342] P 8484181c4ece483c8b408817fab1a917: Request end of tablet copy session 2254b3ee3db647598d781e467584f78d-e51db70f5f284c4da3c1669fddb41b00 received from {username='slave'} at 127.8.86.136:42777
I20240502 14:00:38.450870 15721 tablet_copy_service.cc:433] P 8484181c4ece483c8b408817fab1a917: ending tablet copy session 2254b3ee3db647598d781e467584f78d-e51db70f5f284c4da3c1669fddb41b00 on tablet e51db70f5f284c4da3c1669fddb41b00 with peer 2254b3ee3db647598d781e467584f78d
I20240502 14:00:38.450968 19001 raft_consensus.cc:348] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:38.451151 19001 raft_consensus.cc:729] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f688370715454d6e814a78fcf85ac5c3, State: Initialized, Role: LEARNER
I20240502 14:00:38.451306 19001 consensus_queue.cc:260] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:38.451702 19001 ts_tablet_manager.cc:1398] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:38.452358 15721 tablet_copy_service.cc:342] P 8484181c4ece483c8b408817fab1a917: Request end of tablet copy session f688370715454d6e814a78fcf85ac5c3-e51db70f5f284c4da3c1669fddb41b00 received from {username='slave'} at 127.8.86.133:54505
I20240502 14:00:38.452440 15721 tablet_copy_service.cc:433] P 8484181c4ece483c8b408817fab1a917: ending tablet copy session f688370715454d6e814a78fcf85ac5c3-e51db70f5f284c4da3c1669fddb41b00 on tablet e51db70f5f284c4da3c1669fddb41b00 with peer f688370715454d6e814a78fcf85ac5c3
I20240502 14:00:38.459990 19380 tablet_bootstrap.cc:492] T b6d365c10c76416a944ec8fe409e3dc5 P 2254b3ee3db647598d781e467584f78d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:38.463124 19380 tablet_bootstrap.cc:492] T b6d365c10c76416a944ec8fe409e3dc5 P 2254b3ee3db647598d781e467584f78d: Bootstrap complete.
I20240502 14:00:38.463313 19380 ts_tablet_manager.cc:1367] T b6d365c10c76416a944ec8fe409e3dc5 P 2254b3ee3db647598d781e467584f78d: Time spent bootstrapping tablet: real 0.156s user 0.026s sys 0.000s
I20240502 14:00:38.463778 19380 raft_consensus.cc:348] T b6d365c10c76416a944ec8fe409e3dc5 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:38.465649 15416 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c004727e7c64f33bc9492184b77fde9 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:38.463898 19380 raft_consensus.cc:729] T b6d365c10c76416a944ec8fe409e3dc5 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2254b3ee3db647598d781e467584f78d, State: Initialized, Role: LEARNER
I20240502 14:00:38.466363 19380 consensus_queue.cc:260] T b6d365c10c76416a944ec8fe409e3dc5 P 2254b3ee3db647598d781e467584f78d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:38.466712 19380 ts_tablet_manager.cc:1398] T b6d365c10c76416a944ec8fe409e3dc5 P 2254b3ee3db647598d781e467584f78d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:38.467854 15721 tablet_copy_service.cc:342] P 8484181c4ece483c8b408817fab1a917: Request end of tablet copy session 2254b3ee3db647598d781e467584f78d-b6d365c10c76416a944ec8fe409e3dc5 received from {username='slave'} at 127.8.86.136:42777
I20240502 14:00:38.467938 15721 tablet_copy_service.cc:433] P 8484181c4ece483c8b408817fab1a917: ending tablet copy session 2254b3ee3db647598d781e467584f78d-b6d365c10c76416a944ec8fe409e3dc5 on tablet b6d365c10c76416a944ec8fe409e3dc5 with peer 2254b3ee3db647598d781e467584f78d
I20240502 14:00:38.470023 19222 ts_tablet_manager.cc:1888] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:00:38.470160 19222 log.cc:1196] T 03dd735113ab4ea6bdb696b11a4efd7f P 8484181c4ece483c8b408817fab1a917: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/wals/03dd735113ab4ea6bdb696b11a4efd7f
I20240502 14:00:38.470738 15413 catalog_manager.cc:4830] TS 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): tablet 03dd735113ab4ea6bdb696b11a4efd7f (table rebalance_test_table_7 [id=fc947d3d4f264291a81e7e95cb84f48a]) successfully deleted
I20240502 14:00:38.476768 19535 tablet_bootstrap.cc:492] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:38.477069 19535 tablet_bootstrap.cc:492] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: Bootstrap complete.
I20240502 14:00:38.477239 19535 ts_tablet_manager.cc:1367] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: Time spent bootstrapping tablet: real 0.076s user 0.021s sys 0.004s
I20240502 14:00:38.477648 19535 raft_consensus.cc:348] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:38.477771 19535 raft_consensus.cc:729] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2254b3ee3db647598d781e467584f78d, State: Initialized, Role: LEARNER
I20240502 14:00:38.477866 19535 consensus_queue.cc:260] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:38.478122 19535 ts_tablet_manager.cc:1398] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:38.478641 15591 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session 2254b3ee3db647598d781e467584f78d-d81f14b7c1d1468196d0fe6d67fe93a7 received from {username='slave'} at 127.8.86.136:58333
I20240502 14:00:38.478719 15591 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session 2254b3ee3db647598d781e467584f78d-d81f14b7c1d1468196d0fe6d67fe93a7 on tablet d81f14b7c1d1468196d0fe6d67fe93a7 with peer 2254b3ee3db647598d781e467584f78d
I20240502 14:00:38.489745 15832 consensus_queue.cc:237] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:38.490025 15964 consensus_queue.cc:237] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:38.491792 17511 raft_consensus.cc:1261] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.491892 15701 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:38.492677 18507 consensus_queue.cc:1035] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:00:38.492833 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9391f16ae2d247ec8a71e5c971cdda71. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:38.492930 18566 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:38.494953 15701 raft_consensus.cc:1261] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.495616 15832 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:38.495577 18507 consensus_queue.cc:1035] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:38.496500 18566 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:38.498358 15964 raft_consensus.cc:1261] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.498587 18507 raft_consensus.cc:2932] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.499101 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:38.499086 15701 raft_consensus.cc:2932] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.500103 18507 consensus_queue.cc:1035] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:38.502377 17511 raft_consensus.cc:2932] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.502522 15964 raft_consensus.cc:2932] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.505100 15416 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7c57cab68b764db0917ed98e74757ff0 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:38.505265 15429 catalog_manager.cc:5484] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 6 to 7, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.508545 17627 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c57cab68b764db0917ed98e74757ff0 with delete_type TABLET_DATA_TOMBSTONED (TS 2254b3ee3db647598d781e467584f78d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37604
I20240502 14:00:38.508746 19441 tablet_replica.cc:330] T 7c57cab68b764db0917ed98e74757ff0 P 2254b3ee3db647598d781e467584f78d: stopping tablet replica
I20240502 14:00:38.508896 19441 raft_consensus.cc:2229] T 7c57cab68b764db0917ed98e74757ff0 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Raft consensus shutting down.
I20240502 14:00:38.509092 19441 raft_consensus.cc:2258] T 7c57cab68b764db0917ed98e74757ff0 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Raft consensus is shut down!
I20240502 14:00:38.514851 19441 ts_tablet_manager.cc:1875] T 7c57cab68b764db0917ed98e74757ff0 P 2254b3ee3db647598d781e467584f78d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.516924 19441 ts_tablet_manager.cc:1888] T 7c57cab68b764db0917ed98e74757ff0 P 2254b3ee3db647598d781e467584f78d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:00:38.516999 19441 log.cc:1196] T 7c57cab68b764db0917ed98e74757ff0 P 2254b3ee3db647598d781e467584f78d: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-7/wal/wals/7c57cab68b764db0917ed98e74757ff0
I20240502 14:00:38.517566 15413 catalog_manager.cc:4830] TS 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): tablet 7c57cab68b764db0917ed98e74757ff0 (table rebalance_test_table_8 [id=a148b6d170714c99a6024d0a3efa85c8]) successfully deleted
I20240502 14:00:38.524435 19285 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:38.524603 17511 raft_consensus.cc:1203] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:00:38.527958 15700 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:38.529412 15423 catalog_manager.cc:5484] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:38.531121 19482 tablet_bootstrap.cc:492] T 837e8b1235a44131bdaef24b763d1cac P 2254b3ee3db647598d781e467584f78d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:38.531410 19482 tablet_bootstrap.cc:492] T 837e8b1235a44131bdaef24b763d1cac P 2254b3ee3db647598d781e467584f78d: Bootstrap complete.
I20240502 14:00:38.531608 19482 ts_tablet_manager.cc:1367] T 837e8b1235a44131bdaef24b763d1cac P 2254b3ee3db647598d781e467584f78d: Time spent bootstrapping tablet: real 0.210s user 0.027s sys 0.004s
I20240502 14:00:38.532075 19482 raft_consensus.cc:348] T 837e8b1235a44131bdaef24b763d1cac P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:38.532189 19482 raft_consensus.cc:729] T 837e8b1235a44131bdaef24b763d1cac P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2254b3ee3db647598d781e467584f78d, State: Initialized, Role: LEARNER
I20240502 14:00:38.532282 19482 consensus_queue.cc:260] T 837e8b1235a44131bdaef24b763d1cac P 2254b3ee3db647598d781e467584f78d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:38.532675 19482 ts_tablet_manager.cc:1398] T 837e8b1235a44131bdaef24b763d1cac P 2254b3ee3db647598d781e467584f78d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:38.533882 15984 tablet_copy_service.cc:342] P 4d0f8f311e834d81bd54b9f8e7923bcd: Request end of tablet copy session 2254b3ee3db647598d781e467584f78d-837e8b1235a44131bdaef24b763d1cac received from {username='slave'} at 127.8.86.136:55499
I20240502 14:00:38.533980 15984 tablet_copy_service.cc:433] P 4d0f8f311e834d81bd54b9f8e7923bcd: ending tablet copy session 2254b3ee3db647598d781e467584f78d-837e8b1235a44131bdaef24b763d1cac on tablet 837e8b1235a44131bdaef24b763d1cac with peer 2254b3ee3db647598d781e467584f78d
I20240502 14:00:38.537253 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:38.540758 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:38.549649 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:38.561838 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:38.565066 15832 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:38.570319 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:38.573062 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:38.584324 15568 consensus_queue.cc:237] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
W20240502 14:00:38.586171 15506 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 436c127af87c49e8b7bf9b6571f7dc30. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:38.588059 17381 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:38.588685 15700 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:38.588997 18653 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:38.589574 18653 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:38.608064 17251 raft_consensus.cc:1203] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:00:38.625694 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:38.636337 17646 raft_consensus.cc:1203] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:00:38.642688 18653 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:38.645310 15700 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:38.645725 17381 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:38.646848 15424 catalog_manager.cc:5484] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 8 to 9, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:38.653939 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:38.656347 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:38.662637 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:38.670888 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:38.672607 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:38.678586 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:38.692075 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:38.697080 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:38.697209 17646 raft_consensus.cc:1203] T b6d365c10c76416a944ec8fe409e3dc5 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:38.700562 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:38.715252 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:38.716153 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:38.733131 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:38.734027 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:38.737690 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:38.760269 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:38.764358 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:38.784763 15964 consensus_queue.cc:237] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:38.786139 17381 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:38.786500 17251 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:38.786708 18566 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:38.786870 18418 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:38.787246 15568 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:38.787631 18566 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:38.789880 18418 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.790262 15568 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.793329 17251 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.793362 15424 catalog_manager.cc:5484] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 8 to 9. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:38.795327 17381 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.800092 15964 consensus_queue.cc:237] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:38.801427 17381 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:38.801970 18566 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:38.802112 17251 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:38.803845 19218 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:38.804235 19218 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.804980 17381 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.806948 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 387bdd8ae3b34f588e2170b061a5bc05 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:38.807051 15422 catalog_manager.cc:5484] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 9 to 10, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:38.808903 17251 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.808996 15964 consensus_queue.cc:237] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:38.810165 15549 tablet_service.cc:1514] Processing DeleteTablet for tablet 387bdd8ae3b34f588e2170b061a5bc05 with delete_type TABLET_DATA_TOMBSTONED (TS 1eb0e3a6c782469fa8110f6bb6a585cf not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44898
I20240502 14:00:38.811450 17511 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:38.811760 19665 tablet_replica.cc:330] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf: stopping tablet replica
I20240502 14:00:38.811815 17646 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:38.811967 19665 raft_consensus.cc:2229] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.812127 19665 raft_consensus.cc:2258] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.812252 15568 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 14:00:38.812917 15898 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e48dda3e62e14e46b03d79648e3e2773. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:38.813213 18418 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:38.813743 19665 ts_tablet_manager.cc:1875] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.814656 19218 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:38.814894 15964 consensus_queue.cc:237] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:38.816174 17251 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:00:38.816764 17381 raft_consensus.cc:1261] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:00:38.816917 19218 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:38.817148 18418 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 14:00:38.817857 15897 consensus_peers.cc:485] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 387bdd8ae3b34f588e2170b061a5bc05. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:38.818066 18418 consensus_queue.cc:1035] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:00:38.818948 19665 ts_tablet_manager.cc:1888] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 14:00:38.819082 19665 log.cc:1196] T 387bdd8ae3b34f588e2170b061a5bc05 P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/wals/387bdd8ae3b34f588e2170b061a5bc05
I20240502 14:00:38.819466 17251 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:38.819696 18566 raft_consensus.cc:1050] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd: attempting to promote NON_VOTER 8fea283177394bd28d7eacf4fb744d59 to VOTER
I20240502 14:00:38.819705 15415 catalog_manager.cc:4830] TS 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237): tablet 387bdd8ae3b34f588e2170b061a5bc05 (table rebalance_test_table_1 [id=a00d0c0f593149dcb05a21e3d90dda57]) successfully deleted
I20240502 14:00:38.820253 18566 consensus_queue.cc:237] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:38.821038 19218 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:38.821882 18566 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.821961 17646 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.823840 15413 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 387bdd8ae3b34f588e2170b061a5bc05 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:00:38.824229 17381 raft_consensus.cc:2932] T 387bdd8ae3b34f588e2170b061a5bc05 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:38.824492 15700 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:38.824555 15422 catalog_manager.cc:5484] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d reported cstate change: config changed from index 7 to 8, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.825048 19285 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.826674 15420 catalog_manager.cc:5484] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 reported cstate change: config changed from index 10 to 11, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:38.827587 17514 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:38.828210 15567 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:38.828475 19285 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.829267 19285 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.831004 17251 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:38.831496 19285 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.832666 17511 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.833119 15568 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.834981 19674 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.837103 15568 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.838196 17511 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.838784 15422 catalog_manager.cc:5484] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:38.842903 18566 raft_consensus.cc:1050] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd: attempting to promote NON_VOTER f688370715454d6e814a78fcf85ac5c3 to VOTER
I20240502 14:00:38.843448 19285 raft_consensus.cc:1050] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd: attempting to promote NON_VOTER 8fea283177394bd28d7eacf4fb744d59 to VOTER
I20240502 14:00:38.843472 18566 consensus_queue.cc:237] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:38.843811 19285 consensus_queue.cc:237] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:38.842950 18418 raft_consensus.cc:1050] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd: attempting to promote NON_VOTER 8fea283177394bd28d7eacf4fb744d59 to VOTER
W20240502 14:00:38.845355 18418 raft_consensus.cc:1053] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd: Unable to promote non-voter 8fea283177394bd28d7eacf4fb744d59: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:38.845428 17646 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:38.845278 17250 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.847033 15568 consensus_queue.cc:237] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
W20240502 14:00:38.847332 15898 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e48dda3e62e14e46b03d79648e3e2773. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:38.848158 15567 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:38.848555 19218 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:38.848834 19218 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:38.850070 17381 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:38.850450 15701 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:38.851020 19048 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:38.851145 18653 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 14:00:38.851683 15506 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 436c127af87c49e8b7bf9b6571f7dc30. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:38.852015 17511 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:38.852403 18418 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:38.854353 19320 raft_consensus.cc:1050] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf: attempting to promote NON_VOTER 8fea283177394bd28d7eacf4fb744d59 to VOTER
I20240502 14:00:38.854820 19320 consensus_queue.cc:237] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:38.856428 15567 raft_consensus.cc:1261] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:38.856705 17511 raft_consensus.cc:1261] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:38.856858 18506 consensus_queue.cc:1035] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:38.857077 19321 consensus_queue.cc:1035] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:38.857707 15964 raft_consensus.cc:1261] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:38.859400 19321 consensus_queue.cc:1035] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:38.859933 19321 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.860661 17248 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.861109 19218 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:38.861518 17646 raft_consensus.cc:1203] T 837e8b1235a44131bdaef24b763d1cac P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:00:38.862802 15567 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.864876 18418 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.865116 17511 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.865866 15422 catalog_manager.cc:5484] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 4 to 5, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:38.866544 19218 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:38.867377 17515 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
W20240502 14:00:38.867707 15504 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 436c127af87c49e8b7bf9b6571f7dc30. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:38.868607 15698 consensus_queue.cc:237] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:38.868692 18653 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:38.870095 15702 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.871937 17515 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:38.871973 17514 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.872071 15424 catalog_manager.cc:5484] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 8 to 9, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:38.872476 17646 raft_consensus.cc:1203] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:00:38.874033 18605 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:38.874106 15571 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:38.874667 17380 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:38.875141 18418 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
W20240502 14:00:38.875293 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7981c3f46d8483caae82fbf2296e762. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:38.875639 15964 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.876750 19218 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.876935 15960 consensus_queue.cc:237] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:38.877647 17248 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.878185 17646 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.878505 18605 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:38.879334 17511 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:38.879631 15571 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:38.879674 19218 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:38.880254 15421 catalog_manager.cc:5484] T 436c127af87c49e8b7bf9b6571f7dc30 P 3c3658c946994e2ba9f41b64aa327d25 reported cstate change: config changed from index 9 to 10, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:38.880383 19674 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:38.880440 15702 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:38.881559 15426 catalog_manager.cc:5484] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 reported cstate change: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.882391 15700 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.883913 15700 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.884006 15831 consensus_queue.cc:237] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:38.885033 15567 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.885934 18502 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:38.887475 17515 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.887789 15571 raft_consensus.cc:1261] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:38.887850 17512 raft_consensus.cc:1261] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:38.889168 19332 consensus_queue.cc:1035] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.889482 19671 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:38.889544 19332 consensus_queue.cc:1035] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:38.890410 15571 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:38.892120 17515 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:38.892426 17646 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:38.892900 18418 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:38.895938 19332 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.899243 15416 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 28e371c1a999431499e09479e8f5b44b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:38.900086 15424 catalog_manager.cc:5484] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 5 to 6, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:38.900487 18418 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.900600 17515 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.902330 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e48dda3e62e14e46b03d79648e3e2773 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:38.902752 17646 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.904192 15421 catalog_manager.cc:5484] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 reported cstate change: config changed from index 8 to 9, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:38.904891 17515 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.906378 15944 tablet_service.cc:1514] Processing DeleteTablet for tablet 28e371c1a999431499e09479e8f5b44b with delete_type TABLET_DATA_TOMBSTONED (TS 4d0f8f311e834d81bd54b9f8e7923bcd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56876
I20240502 14:00:38.906843 15571 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:38.909030 15421 catalog_manager.cc:5484] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59 reported cstate change: config changed from index 9 to 10, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.913203 15960 consensus_queue.cc:237] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:38.913928 17231 tablet_service.cc:1514] Processing DeleteTablet for tablet e48dda3e62e14e46b03d79648e3e2773 with delete_type TABLET_DATA_TOMBSTONED (TS f688370715454d6e814a78fcf85ac5c3 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:60394
I20240502 14:00:38.914070 19715 tablet_replica.cc:330] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd: stopping tablet replica
I20240502 14:00:38.914203 19715 raft_consensus.cc:2229] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.914472 19715 raft_consensus.cc:2258] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.915199 17646 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:00:38.915351 19715 ts_tablet_manager.cc:1875] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.915804 19671 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:00:38.917342 19673 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.917654 17646 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.918154 19715 ts_tablet_manager.cc:1888] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:00:38.918249 19715 log.cc:1196] T 28e371c1a999431499e09479e8f5b44b P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal/wals/28e371c1a999431499e09479e8f5b44b
I20240502 14:00:38.919054 15413 catalog_manager.cc:4830] TS 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719): tablet 28e371c1a999431499e09479e8f5b44b (table rebalance_test_table_3 [id=12389884a03b40f68c532a06ec761e9c]) successfully deleted
I20240502 14:00:38.919240 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e48dda3e62e14e46b03d79648e3e2773 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:38.919476 15960 consensus_queue.cc:237] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:38.919684 15567 raft_consensus.cc:1203] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.10] Dedup: 1.9->[1.10-1.10]
I20240502 14:00:38.919845 15567 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.921746 15421 catalog_manager.cc:5484] T e48dda3e62e14e46b03d79648e3e2773 P 2254b3ee3db647598d781e467584f78d reported cstate change: config changed from index 10 to 11, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
W20240502 14:00:38.922688 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d92e5885bf94c3e8e0e4dee40017699. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:38.922853 17514 raft_consensus.cc:1261] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.923472 19672 consensus_queue.cc:1035] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:38.924011 17515 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:00:38.924573 15831 raft_consensus.cc:1261] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.925096 15549 tablet_service.cc:1514] Processing DeleteTablet for tablet e48dda3e62e14e46b03d79648e3e2773 with delete_type TABLET_DATA_TOMBSTONED (TS 1eb0e3a6c782469fa8110f6bb6a585cf not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:44898
I20240502 14:00:38.925284 19665 tablet_replica.cc:330] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf: stopping tablet replica
I20240502 14:00:38.925398 19665 raft_consensus.cc:2229] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.925616 19665 pending_rounds.cc:62] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf: Trying to abort 1 pending ops.
I20240502 14:00:38.925681 19665 pending_rounds.cc:68] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf: Aborting op as it isn't in flight: id { term: 1 index: 10 } timestamp: 7023240965638909952 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "e48dda3e62e14e46b03d79648e3e2773" old_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } } new_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } } }
I20240502 14:00:38.925812 19665 raft_consensus.cc:2894] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Aborting config change with OpId 1.10: Aborted: Op aborted
I20240502 14:00:38.925863 19665 raft_consensus.cc:2866] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:00:38.925900 19665 raft_consensus.cc:2258] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.925714 18418 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:38.926602 19672 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:00:38.927403 17514 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.929137 17514 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:38.930176 19671 consensus_queue.cc:1035] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:38.930868 15424 catalog_manager.cc:5484] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:38.931027 15569 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.931207 15831 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
W20240502 14:00:38.933689 19668 consensus_queue.cc:1159] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" status { last_received { term: 1 index: 10 } last_committed_idx: 9 last_received_current_leader { term: 1 index: 10 } }
I20240502 14:00:38.934896 19665 ts_tablet_manager.cc:1875] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.936962 19665 ts_tablet_manager.cc:1888] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 14:00:38.937044 19665 log.cc:1196] T e48dda3e62e14e46b03d79648e3e2773 P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/wals/e48dda3e62e14e46b03d79648e3e2773
I20240502 14:00:38.937680 15415 catalog_manager.cc:4830] TS 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237): tablet e48dda3e62e14e46b03d79648e3e2773 (table rebalance_test_table_9 [id=17d53f89ea7b4f918ed8bb35d44f6401]) successfully deleted
W20240502 14:00:38.943676 15414 catalog_manager.cc:4797] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): delete failed for tablet e48dda3e62e14e46b03d79648e3e2773 because the tablet was not found. No further retry: Not found: Tablet not found: e48dda3e62e14e46b03d79648e3e2773
I20240502 14:00:38.953231 15831 consensus_queue.cc:237] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:00:38.954932 15831 consensus_queue.cc:237] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:38.955257 17515 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20240502 14:00:38.957093 15766 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): Couldn't send request to peer 8484181c4ece483c8b408817fab1a917. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:38.957198 19320 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:00:38.957299 17248 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 14:00:38.957666 15960 raft_consensus.cc:1261] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:38.957887 17515 raft_consensus.cc:1261] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:38.958231 19320 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:00:38.958271 19332 consensus_queue.cc:1035] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:38.959538 19320 consensus_queue.cc:1035] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:38.960379 15702 consensus_queue.cc:237] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:38.960460 19320 raft_consensus.cc:2932] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.961769 15567 raft_consensus.cc:1261] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:00:38.961887 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f80c8590e994a10a9b7c6c174e28173. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:38.962249 18174 consensus_queue.cc:1035] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:38.962646 17380 raft_consensus.cc:1261] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:38.962613 15960 raft_consensus.cc:2932] T 7c57cab68b764db0917ed98e74757ff0 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.962949 15416 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7c57cab68b764db0917ed98e74757ff0 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:38.964223 17515 raft_consensus.cc:2932] T 7c57cab68b764db0917ed98e74757ff0 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:38.964395 15421 catalog_manager.cc:5484] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:38.966972 18174 consensus_queue.cc:1035] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:38.968997 15681 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c57cab68b764db0917ed98e74757ff0 with delete_type TABLET_DATA_TOMBSTONED (TS 8484181c4ece483c8b408817fab1a917 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:38324
I20240502 14:00:38.969210 19222 tablet_replica.cc:330] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917: stopping tablet replica
I20240502 14:00:38.969357 19222 raft_consensus.cc:2229] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:38.969553 19222 raft_consensus.cc:2258] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:38.970270 19222 ts_tablet_manager.cc:1875] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:38.972481 19222 ts_tablet_manager.cc:1888] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:00:38.972541 19222 log.cc:1196] T 7c57cab68b764db0917ed98e74757ff0 P 8484181c4ece483c8b408817fab1a917: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/wals/7c57cab68b764db0917ed98e74757ff0
I20240502 14:00:38.973456 15413 catalog_manager.cc:4830] TS 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): tablet 7c57cab68b764db0917ed98e74757ff0 (table rebalance_test_table_8 [id=a148b6d170714c99a6024d0a3efa85c8]) successfully deleted
I20240502 14:00:38.976642 18605 raft_consensus.cc:2932] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.977833 15567 raft_consensus.cc:2932] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.979346 17380 raft_consensus.cc:2932] T 1f80c8590e994a10a9b7c6c174e28173 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:38.979781 19332 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } } }
I20240502 14:00:38.982728 15424 catalog_manager.cc:5484] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 6 to 7, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:38.982975 19743 ts_tablet_manager.cc:902] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25: Initiating tablet copy from peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:38.983454 19743 tablet_copy_client.cc:308] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.132:45719
I20240502 14:00:38.983686 15420 catalog_manager.cc:5484] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 12 to 13, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:38.984409 17514 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } } }
I20240502 14:00:38.984997 17250 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } } }
I20240502 14:00:38.985368 15984 tablet_copy_service.cc:140] P 4d0f8f311e834d81bd54b9f8e7923bcd: Received BeginTabletCopySession request for tablet 9391f16ae2d247ec8a71e5c971cdda71 from peer 3c3658c946994e2ba9f41b64aa327d25 ({username='slave'} at 127.8.86.134:37349)
I20240502 14:00:38.985489 15984 tablet_copy_service.cc:161] P 4d0f8f311e834d81bd54b9f8e7923bcd: Beginning new tablet copy session on tablet 9391f16ae2d247ec8a71e5c971cdda71 from peer 3c3658c946994e2ba9f41b64aa327d25 at {username='slave'} at 127.8.86.134:37349: session id = 3c3658c946994e2ba9f41b64aa327d25-9391f16ae2d247ec8a71e5c971cdda71
I20240502 14:00:38.986452 15984 tablet_copy_source_session.cc:205] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:38.988593 19743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9391f16ae2d247ec8a71e5c971cdda71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.990815 19743 tablet_copy_client.cc:783] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:38.990978 19743 tablet_copy_client.cc:647] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:38.994762 19743 tablet_copy_client.cc:523] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:38.996174 19743 tablet_bootstrap.cc:492] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap starting.
I20240502 14:00:39.001266 15567 consensus_queue.cc:237] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
W20240502 14:00:39.002859 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0cd21dc897b41aca9118be6df48cc0c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:39.003718 17380 raft_consensus.cc:1261] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:39.003937 15960 raft_consensus.cc:1261] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:39.004323 18653 consensus_queue.cc:1035] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:39.004629 19694 consensus_queue.cc:1035] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:39.007679 18653 raft_consensus.cc:2932] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:39.009589 15831 consensus_queue.cc:237] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:39.010391 15960 raft_consensus.cc:2932] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:39.010762 15426 catalog_manager.cc:5484] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 6 to 7, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:39.011670 15964 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:39.012279 17248 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:39.012708 16899 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 14:00:39.013252 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2886dfbec1e141198b0cf7a435fac498. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:39.013945 17380 raft_consensus.cc:2932] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:39.013267 16899 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
I20240502 14:00:39.017701 16899 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:39.020733 17248 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:39.021296 15426 catalog_manager.cc:5484] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 8 to 9, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:39.023685 15964 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:00:39.026475 19756 raft_consensus.cc:1050] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917: attempting to promote NON_VOTER 2254b3ee3db647598d781e467584f78d to VOTER
I20240502 14:00:39.026917 19756 consensus_queue.cc:237] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:39.028847 15567 raft_consensus.cc:1261] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:39.029307 18605 consensus_queue.cc:1035] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:39.030715 15964 raft_consensus.cc:1261] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:39.031330 17646 raft_consensus.cc:1261] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:39.032655 18174 consensus_queue.cc:1035] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:39.032919 18174 consensus_queue.cc:1035] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:39.036005 19743 tablet_bootstrap.cc:492] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:39.036351 17248 raft_consensus.cc:1261] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:39.036590 19743 tablet_bootstrap.cc:492] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap complete.
I20240502 14:00:39.036834 19743 ts_tablet_manager.cc:1367] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25: Time spent bootstrapping tablet: real 0.041s user 0.029s sys 0.000s
I20240502 14:00:39.036806 18605 consensus_queue.cc:1035] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:39.037390 19743 raft_consensus.cc:348] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:39.037586 19743 raft_consensus.cc:729] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c3658c946994e2ba9f41b64aa327d25, State: Initialized, Role: LEARNER
I20240502 14:00:39.037735 19743 consensus_queue.cc:260] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:39.038654 19743 ts_tablet_manager.cc:1398] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:39.039147 15984 tablet_copy_service.cc:342] P 4d0f8f311e834d81bd54b9f8e7923bcd: Request end of tablet copy session 3c3658c946994e2ba9f41b64aa327d25-9391f16ae2d247ec8a71e5c971cdda71 received from {username='slave'} at 127.8.86.134:37349
I20240502 14:00:39.039247 15984 tablet_copy_service.cc:433] P 4d0f8f311e834d81bd54b9f8e7923bcd: ending tablet copy session 3c3658c946994e2ba9f41b64aa327d25-9391f16ae2d247ec8a71e5c971cdda71 on tablet 9391f16ae2d247ec8a71e5c971cdda71 with peer 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:39.044364 18605 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:39.047400 15567 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:39.047890 17646 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:39.050367 15964 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:39.051205 18174 raft_consensus.cc:1050] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917: attempting to promote NON_VOTER f688370715454d6e814a78fcf85ac5c3 to VOTER
I20240502 14:00:39.051681 18174 consensus_queue.cc:237] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:39.053046 15428 catalog_manager.cc:5484] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:39.053805 15964 raft_consensus.cc:1261] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:39.053946 17248 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:39.054271 17646 raft_consensus.cc:1261] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:39.056602 18605 consensus_queue.cc:1035] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:39.056870 18605 consensus_queue.cc:1035] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:39.057812 17250 raft_consensus.cc:1261] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:39.058544 19756 consensus_queue.cc:1035] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:39.060263 15567 raft_consensus.cc:1261] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:39.060689 19756 consensus_queue.cc:1035] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:39.063071 19756 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:39.065213 15567 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:39.067749 15420 catalog_manager.cc:5484] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:39.068048 17646 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:39.068671 15964 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:39.074275 17248 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:39.079334 15702 consensus_queue.cc:237] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:39.080924 17248 raft_consensus.cc:1261] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:39.081279 17646 raft_consensus.cc:1261] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:39.081423 19756 consensus_queue.cc:1035] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:39.081672 15964 raft_consensus.cc:1261] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:39.081761 19756 consensus_queue.cc:1035] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:39.082368 18174 consensus_queue.cc:1035] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:39.084059 18502 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:39.084439 15964 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:39.085148 17248 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:39.086712 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e51db70f5f284c4da3c1669fddb41b00 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:39.087628 15429 catalog_manager.cc:5484] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:39.088380 17646 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:39.092825 15549 tablet_service.cc:1514] Processing DeleteTablet for tablet e51db70f5f284c4da3c1669fddb41b00 with delete_type TABLET_DATA_TOMBSTONED (TS 1eb0e3a6c782469fa8110f6bb6a585cf not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:44898
I20240502 14:00:39.093086 19665 tablet_replica.cc:330] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf: stopping tablet replica
I20240502 14:00:39.093227 19665 raft_consensus.cc:2229] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:39.093433 19665 raft_consensus.cc:2258] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:39.094571 19665 ts_tablet_manager.cc:1875] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:39.096877 19665 ts_tablet_manager.cc:1888] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:00:39.096946 19665 log.cc:1196] T e51db70f5f284c4da3c1669fddb41b00 P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/wals/e51db70f5f284c4da3c1669fddb41b00
I20240502 14:00:39.097627 15415 catalog_manager.cc:4830] TS 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237): tablet e51db70f5f284c4da3c1669fddb41b00 (table rebalance_test_table_8 [id=a148b6d170714c99a6024d0a3efa85c8]) successfully deleted
I20240502 14:00:39.100562 15702 consensus_queue.cc:237] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:39.102696 17248 raft_consensus.cc:1261] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:39.103569 17646 raft_consensus.cc:1261] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:39.103719 19794 consensus_queue.cc:1035] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:39.104259 19756 consensus_queue.cc:1035] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:39.112419 19756 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:39.113916 17646 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:39.114158 17250 raft_consensus.cc:2932] T e51db70f5f284c4da3c1669fddb41b00 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:39.116318 15964 consensus_queue.cc:237] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:39.116815 15426 catalog_manager.cc:5484] T e51db70f5f284c4da3c1669fddb41b00 P 2254b3ee3db647598d781e467584f78d reported cstate change: config changed from index 8 to 9, VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:39.118142 17380 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:39.118136 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e51db70f5f284c4da3c1669fddb41b00 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:39.118512 15702 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:39.119182 19673 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:39.119300 18418 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:39.120028 15831 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:39.120442 19673 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:00:39.121057 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9391f16ae2d247ec8a71e5c971cdda71. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:39.122442 19671 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:39.124843 15831 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:39.125335 15944 tablet_service.cc:1514] Processing DeleteTablet for tablet e51db70f5f284c4da3c1669fddb41b00 with delete_type TABLET_DATA_TOMBSTONED (TS 4d0f8f311e834d81bd54b9f8e7923bcd not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:56876
I20240502 14:00:39.125340 15702 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:39.126621 19715 tablet_replica.cc:330] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd: stopping tablet replica
I20240502 14:00:39.126752 19715 raft_consensus.cc:2229] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:39.126974 19715 raft_consensus.cc:2258] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:39.131019 15425 catalog_manager.cc:5484] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:39.132431 19715 ts_tablet_manager.cc:1875] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:39.138859 19715 ts_tablet_manager.cc:1888] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:00:39.140801 19715 log.cc:1196] T e51db70f5f284c4da3c1669fddb41b00 P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-3/wal/wals/e51db70f5f284c4da3c1669fddb41b00
I20240502 14:00:39.141696 15413 catalog_manager.cc:4830] TS 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719): tablet e51db70f5f284c4da3c1669fddb41b00 (table rebalance_test_table_8 [id=a148b6d170714c99a6024d0a3efa85c8]) successfully deleted
I20240502 14:00:39.142678 19673 raft_consensus.cc:1050] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd: attempting to promote NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 to VOTER
I20240502 14:00:39.143174 19673 consensus_queue.cc:237] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:39.144191 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:39.145084 15831 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:39.145814 19218 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:39.146677 17381 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:39.147884 15702 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:39.147980 17380 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
W20240502 14:00:39.150374 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9391f16ae2d247ec8a71e5c971cdda71. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:39.153631 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:39.154119 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:39.154197 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:39.154381 19671 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:39.154706 19671 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:39.157562 19671 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:39.159919 15831 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:39.160444 15426 catalog_manager.cc:5484] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:39.162765 17380 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:39.164708 15702 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:39.167438 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:39.170006 15964 consensus_queue.cc:237] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }
I20240502 14:00:39.172523 15831 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:39.172523 15702 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:39.172909 19218 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:39.173125 17380 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:39.173224 19218 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:39.173866 19674 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:39.176215 19674 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:39.177953 15831 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:39.178498 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9391f16ae2d247ec8a71e5c971cdda71 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:39.178865 15702 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:39.182456 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:39.183292 15425 catalog_manager.cc:5484] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 6 to 7, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:39.186231 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:39.186661 17627 tablet_service.cc:1514] Processing DeleteTablet for tablet 9391f16ae2d247ec8a71e5c971cdda71 with delete_type TABLET_DATA_TOMBSTONED (TS 2254b3ee3db647598d781e467584f78d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37604
I20240502 14:00:39.178969 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:39.212896 17380 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:39.213881 15964 consensus_queue.cc:237] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }
I20240502 14:00:39.215524 15832 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:39.216008 19671 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:39.216545 17380 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:39.217976 19671 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 14:00:39.218523 15413 catalog_manager.cc:4797] TS 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): delete failed for tablet 9391f16ae2d247ec8a71e5c971cdda71 because the tablet was not found. No further retry: Not found: Tablet not found: 9391f16ae2d247ec8a71e5c971cdda71
I20240502 14:00:39.219163 19671 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:39.219751 15567 consensus_queue.cc:237] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:39.221072 17646 raft_consensus.cc:1261] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:39.221709 18402 consensus_queue.cc:1035] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:39.221757 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9391f16ae2d247ec8a71e5c971cdda71 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:39.222123 15832 raft_consensus.cc:1261] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:39.222514 15831 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:39.222628 18402 consensus_queue.cc:1035] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:00:39.223333 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb9f100600a246708457280d91826fce. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:39.226704 17381 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:39.228621 15425 catalog_manager.cc:5484] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 7 to 8, VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) evicted. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:39.232396 15681 tablet_service.cc:1514] Processing DeleteTablet for tablet 9391f16ae2d247ec8a71e5c971cdda71 with delete_type TABLET_DATA_TOMBSTONED (TS 8484181c4ece483c8b408817fab1a917 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:38324
I20240502 14:00:39.232638 19222 tablet_replica.cc:330] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: stopping tablet replica
I20240502 14:00:39.232762 19222 raft_consensus.cc:2229] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:39.232939 19222 raft_consensus.cc:2258] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:39.233330 18402 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:39.233754 19222 ts_tablet_manager.cc:1875] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:39.235695 17646 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:39.236063 19222 ts_tablet_manager.cc:1888] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:00:39.236142 19222 log.cc:1196] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-1/wal/wals/9391f16ae2d247ec8a71e5c971cdda71
I20240502 14:00:39.236189 15425 catalog_manager.cc:5484] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:39.236805 15413 catalog_manager.cc:4830] TS 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): tablet 9391f16ae2d247ec8a71e5c971cdda71 (table rebalance_test_table_0 [id=fcc590955678424ab1c27b0b23f75544]) successfully deleted
I20240502 14:00:39.241205 15832 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:39.247151 19903 ts_tablet_manager.cc:902] T 387bdd8ae3b34f588e2170b061a5bc05 P 2254b3ee3db647598d781e467584f78d: Initiating tablet copy from peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:39.247731 19903 tablet_copy_client.cc:308] T 387bdd8ae3b34f588e2170b061a5bc05 P 2254b3ee3db647598d781e467584f78d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.132:45719
I20240502 14:00:39.248277 15984 tablet_copy_service.cc:140] P 4d0f8f311e834d81bd54b9f8e7923bcd: Received BeginTabletCopySession request for tablet 387bdd8ae3b34f588e2170b061a5bc05 from peer 2254b3ee3db647598d781e467584f78d ({username='slave'} at 127.8.86.136:55499)
I20240502 14:00:39.248376 15984 tablet_copy_service.cc:161] P 4d0f8f311e834d81bd54b9f8e7923bcd: Beginning new tablet copy session on tablet 387bdd8ae3b34f588e2170b061a5bc05 from peer 2254b3ee3db647598d781e467584f78d at {username='slave'} at 127.8.86.136:55499: session id = 2254b3ee3db647598d781e467584f78d-387bdd8ae3b34f588e2170b061a5bc05
I20240502 14:00:39.249423 15984 tablet_copy_source_session.cc:205] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:39.251226 19903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387bdd8ae3b34f588e2170b061a5bc05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.254590 19048 raft_consensus.cc:1050] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf: attempting to promote NON_VOTER 2254b3ee3db647598d781e467584f78d to VOTER
I20240502 14:00:39.255038 19048 consensus_queue.cc:237] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } }
I20240502 14:00:39.255307 19903 tablet_copy_client.cc:783] T 387bdd8ae3b34f588e2170b061a5bc05 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:39.255455 19903 tablet_copy_client.cc:647] T 387bdd8ae3b34f588e2170b061a5bc05 P 2254b3ee3db647598d781e467584f78d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:39.256690 15832 raft_consensus.cc:1261] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:39.256839 17646 raft_consensus.cc:1261] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:39.257105 17514 raft_consensus.cc:1261] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:39.257336 18402 consensus_queue.cc:1035] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:39.258261 18402 consensus_queue.cc:1035] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:39.258618 18653 consensus_queue.cc:1035] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:39.263604 18653 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.263937 17646 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.266274 19903 tablet_copy_client.cc:523] T 387bdd8ae3b34f588e2170b061a5bc05 P 2254b3ee3db647598d781e467584f78d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:39.266506 17514 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.267673 19903 tablet_bootstrap.cc:492] T 387bdd8ae3b34f588e2170b061a5bc05 P 2254b3ee3db647598d781e467584f78d: Bootstrap starting.
I20240502 14:00:39.268291 15426 catalog_manager.cc:5484] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d reported cstate change: config changed from index 7 to 8, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.270740 15832 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.272785 19919 ts_tablet_manager.cc:902] T 436c127af87c49e8b7bf9b6571f7dc30 P 8fea283177394bd28d7eacf4fb744d59: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:39.273353 19919 tablet_copy_client.cc:308] T 436c127af87c49e8b7bf9b6571f7dc30 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:39.273784 15591 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet 436c127af87c49e8b7bf9b6571f7dc30 from peer 8fea283177394bd28d7eacf4fb744d59 ({username='slave'} at 127.8.86.135:41945)
I20240502 14:00:39.273885 15591 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet 436c127af87c49e8b7bf9b6571f7dc30 from peer 8fea283177394bd28d7eacf4fb744d59 at {username='slave'} at 127.8.86.135:41945: session id = 8fea283177394bd28d7eacf4fb744d59-436c127af87c49e8b7bf9b6571f7dc30
I20240502 14:00:39.274839 15591 tablet_copy_source_session.cc:205] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:39.275810 19919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436c127af87c49e8b7bf9b6571f7dc30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.277956 19919 tablet_copy_client.cc:783] T 436c127af87c49e8b7bf9b6571f7dc30 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:39.278100 19919 tablet_copy_client.cc:647] T 436c127af87c49e8b7bf9b6571f7dc30 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:39.282871 15567 consensus_queue.cc:237] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } }
I20240502 14:00:39.283315 19919 tablet_copy_client.cc:523] T 436c127af87c49e8b7bf9b6571f7dc30 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:39.284384 17646 raft_consensus.cc:1261] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:39.284680 19919 tablet_bootstrap.cc:492] T 436c127af87c49e8b7bf9b6571f7dc30 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap starting.
I20240502 14:00:39.284834 18402 consensus_queue.cc:1035] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:39.285648 17514 raft_consensus.cc:1261] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:39.286612 18402 consensus_queue.cc:1035] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:39.288744 19920 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.291200 15415 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d81f14b7c1d1468196d0fe6d67fe93a7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:39.291287 17646 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.294775 15429 catalog_manager.cc:5484] T d81f14b7c1d1468196d0fe6d67fe93a7 P 2254b3ee3db647598d781e467584f78d reported cstate change: config changed from index 8 to 9, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.295686 17514 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.297854 15811 tablet_service.cc:1514] Processing DeleteTablet for tablet d81f14b7c1d1468196d0fe6d67fe93a7 with delete_type TABLET_DATA_TOMBSTONED (TS 82d7b7201cfc431d90baca1ad0a397cf not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:54382
I20240502 14:00:39.302520 19826 raft_consensus.cc:1050] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917: attempting to promote NON_VOTER 2254b3ee3db647598d781e467584f78d to VOTER
I20240502 14:00:39.302953 19826 consensus_queue.cc:237] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } }
I20240502 14:00:39.304826 15832 raft_consensus.cc:1261] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:39.304826 15964 raft_consensus.cc:1261] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:39.305258 18502 consensus_queue.cc:1035] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:39.305624 17646 raft_consensus.cc:1261] T b6d365c10c76416a944ec8fe409e3dc5 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:39.305685 19756 consensus_queue.cc:1035] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:39.307387 19756 consensus_queue.cc:1035] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:39.309923 19756 raft_consensus.cc:2932] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.310616 17646 raft_consensus.cc:2932] T b6d365c10c76416a944ec8fe409e3dc5 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.311900 15964 raft_consensus.cc:2932] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.314440 15429 catalog_manager.cc:5484] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 4 to 5, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:39.318683 15702 consensus_queue.cc:237] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } }
I20240502 14:00:39.318897 19951 tablet_replica.cc:330] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: stopping tablet replica
I20240502 14:00:39.319096 19951 raft_consensus.cc:2229] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:39.320333 19951 raft_consensus.cc:2258] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:39.320937 15964 raft_consensus.cc:1261] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:39.321408 19794 consensus_queue.cc:1035] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:39.323079 19826 raft_consensus.cc:2932] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.323431 19951 ts_tablet_manager.cc:1875] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:39.324895 19919 tablet_bootstrap.cc:492] T 436c127af87c49e8b7bf9b6571f7dc30 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:39.325183 19919 tablet_bootstrap.cc:492] T 436c127af87c49e8b7bf9b6571f7dc30 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap complete.
I20240502 14:00:39.326119 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b6d365c10c76416a944ec8fe409e3dc5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:39.326325 19919 ts_tablet_manager.cc:1367] T 436c127af87c49e8b7bf9b6571f7dc30 P 8fea283177394bd28d7eacf4fb744d59: Time spent bootstrapping tablet: real 0.042s user 0.031s sys 0.000s
I20240502 14:00:39.326611 15425 catalog_manager.cc:5484] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 5 to 6, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:39.327410 19919 raft_consensus.cc:348] T 436c127af87c49e8b7bf9b6571f7dc30 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:39.327693 15964 raft_consensus.cc:2932] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.328358 19919 raft_consensus.cc:729] T 436c127af87c49e8b7bf9b6571f7dc30 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8fea283177394bd28d7eacf4fb744d59, State: Initialized, Role: LEARNER
I20240502 14:00:39.328529 19919 consensus_queue.cc:260] T 436c127af87c49e8b7bf9b6571f7dc30 P 8fea283177394bd28d7eacf4fb744d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:39.330694 19919 ts_tablet_manager.cc:1398] T 436c127af87c49e8b7bf9b6571f7dc30 P 8fea283177394bd28d7eacf4fb744d59: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:39.331238 18418 raft_consensus.cc:1050] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd: attempting to promote NON_VOTER 2254b3ee3db647598d781e467584f78d to VOTER
I20240502 14:00:39.331650 19743 ts_tablet_manager.cc:902] T a7981c3f46d8483caae82fbf2296e762 P 3c3658c946994e2ba9f41b64aa327d25: Initiating tablet copy from peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:39.331636 18418 consensus_queue.cc:237] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } }
I20240502 14:00:39.332062 19743 tablet_copy_client.cc:308] T a7981c3f46d8483caae82fbf2296e762 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:42261
I20240502 14:00:39.332509 15721 tablet_copy_service.cc:140] P 8484181c4ece483c8b408817fab1a917: Received BeginTabletCopySession request for tablet a7981c3f46d8483caae82fbf2296e762 from peer 3c3658c946994e2ba9f41b64aa327d25 ({username='slave'} at 127.8.86.134:44409)
I20240502 14:00:39.332594 15721 tablet_copy_service.cc:161] P 8484181c4ece483c8b408817fab1a917: Beginning new tablet copy session on tablet a7981c3f46d8483caae82fbf2296e762 from peer 3c3658c946994e2ba9f41b64aa327d25 at {username='slave'} at 127.8.86.134:44409: session id = 3c3658c946994e2ba9f41b64aa327d25-a7981c3f46d8483caae82fbf2296e762
I20240502 14:00:39.333597 15811 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d365c10c76416a944ec8fe409e3dc5 with delete_type TABLET_DATA_TOMBSTONED (TS 82d7b7201cfc431d90baca1ad0a397cf not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54382
I20240502 14:00:39.333663 15721 tablet_copy_source_session.cc:205] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:39.334142 17250 raft_consensus.cc:1261] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:39.334983 19743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7981c3f46d8483caae82fbf2296e762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.335165 19951 ts_tablet_manager.cc:1888] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:00:39.335224 19951 log.cc:1196] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/wals/d81f14b7c1d1468196d0fe6d67fe93a7
I20240502 14:00:39.336022 15567 raft_consensus.cc:1261] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:39.336119 15591 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session 8fea283177394bd28d7eacf4fb744d59-436c127af87c49e8b7bf9b6571f7dc30 received from {username='slave'} at 127.8.86.135:41945
I20240502 14:00:39.336171 15591 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session 8fea283177394bd28d7eacf4fb744d59-436c127af87c49e8b7bf9b6571f7dc30 on tablet 436c127af87c49e8b7bf9b6571f7dc30 with peer 8fea283177394bd28d7eacf4fb744d59
I20240502 14:00:39.336453 19285 consensus_queue.cc:1035] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:39.336464 15832 consensus_queue.cc:237] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:39.337172 19743 tablet_copy_client.cc:783] T a7981c3f46d8483caae82fbf2296e762 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:39.337306 19743 tablet_copy_client.cc:647] T a7981c3f46d8483caae82fbf2296e762 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:39.338399 17515 raft_consensus.cc:1261] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:39.338771 17645 raft_consensus.cc:1261] T 837e8b1235a44131bdaef24b763d1cac P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:39.338783 18234 consensus_queue.cc:1035] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:39.338809 15569 raft_consensus.cc:1261] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:39.339223 19218 consensus_queue.cc:1035] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:39.339545 19218 consensus_queue.cc:1035] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:39.340685 17644 raft_consensus.cc:1261] T b6d365c10c76416a944ec8fe409e3dc5 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:39.341171 19321 consensus_queue.cc:1035] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:39.344750 19321 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:39.346235 19826 consensus_queue.cc:1035] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:39.347182 17644 raft_consensus.cc:2932] T b6d365c10c76416a944ec8fe409e3dc5 P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.347517 15569 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:39.350756 15422 catalog_manager.cc:5484] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
W20240502 14:00:39.351184 15766 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28e371c1a999431499e09479e8f5b44b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:39.353642 17514 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } }
I20240502 14:00:39.355414 19951 tablet_replica.cc:330] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf: stopping tablet replica
I20240502 14:00:39.355638 15416 catalog_manager.cc:4830] TS 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531): tablet d81f14b7c1d1468196d0fe6d67fe93a7 (table rebalance_test_table_0 [id=fcc590955678424ab1c27b0b23f75544]) successfully deleted
I20240502 14:00:39.355911 19951 raft_consensus.cc:2229] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:39.357147 19951 pending_rounds.cc:62] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf: Trying to abort 1 pending ops.
I20240502 14:00:39.357203 19951 pending_rounds.cc:68] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023240967383388160 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "b6d365c10c76416a944ec8fe409e3dc5" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } } }
I20240502 14:00:39.357337 19951 raft_consensus.cc:2894] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 14:00:39.357429 19951 raft_consensus.cc:2866] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:00:39.357515 19951 raft_consensus.cc:2258] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:39.358407 19951 ts_tablet_manager.cc:1875] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:39.360873 19951 ts_tablet_manager.cc:1888] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:00:39.360996 19951 log.cc:1196] T b6d365c10c76416a944ec8fe409e3dc5 P 82d7b7201cfc431d90baca1ad0a397cf: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-2/wal/wals/b6d365c10c76416a944ec8fe409e3dc5
I20240502 14:00:39.361531 15416 catalog_manager.cc:4830] TS 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531): tablet b6d365c10c76416a944ec8fe409e3dc5 (table rebalance_test_table_3 [id=12389884a03b40f68c532a06ec761e9c]) successfully deleted
I20240502 14:00:39.362763 19988 ts_tablet_manager.cc:902] T 436c127af87c49e8b7bf9b6571f7dc30 P 2254b3ee3db647598d781e467584f78d: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
F20240502 14:00:39.362996 19988 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-02d3da05d82340178e27218c2ce56b67.queue_time_us, tablet-download-pool-02d3da05d82340178e27218c2ce56b67.queue_time_us) (tablet-download-pool-02d3da05d82340178e27218c2ce56b67.run_cpu_time_us, tablet-download-pool-02d3da05d82340178e27218c2ce56b67.run_cpu_time_us) (tablet-download-pool-02d3da05d82340178e27218c2ce56b67.run_wall_time_us, tablet-download-pool-02d3da05d82340178e27218c2ce56b67.run_wall_time_us) (tablet-download-pool-1c004727e7c64f33bc9492184b77fde9.queue_time_us, tablet-download-pool-1c004727e7c64f33bc9492184b77fde9.queue_time_us) (tablet-download-pool-1c004727e7c64f33bc9492184b77fde9.run_cpu_time_us, tablet-download-pool-1c004727e7c64f33bc9492184b77fde9.run_cpu_time_us) (tablet-download-pool-1c004727e7c64f33bc9492184b77fde9.run_wall_time_us, tablet-download-pool-1c004727e7c64f33bc9492184b77fde9.run_wall_time_us) (tablet-download-pool-1f562ad00ec34a8d9184671ed55d84d1.queue_time_us, tablet-download-pool-1f562ad00ec34a8d9184671ed55d84d1.queue_time_us) (tablet-download-pool-1f562ad00ec34a8d9184671ed55d84d1.run_cpu_time_us, tablet-download-pool-1f562ad00ec34a8d9184671ed55d84d1.run_cpu_time_us) (tablet-download-pool-1f562ad00ec34a8d9184671ed55d84d1.run_wall_time_us, tablet-download-pool-1f562ad00ec34a8d9184671ed55d84d1.run_wall_time_us) (tablet-download-pool-22eef55211ca4f65979022d2f4ae829f.queue_time_us, tablet-download-pool-22eef55211ca4f65979022d2f4ae829f.queue_time_us) (tablet-download-pool-22eef55211ca4f65979022d2f4ae829f.run_cpu_time_us, tablet-download-pool-22eef55211ca4f65979022d2f4ae829f.run_cpu_time_us) (tablet-download-pool-22eef55211ca4f65979022d2f4ae829f.run_wall_time_us, tablet-download-pool-22eef55211ca4f65979022d2f4ae829f.run_wall_time_us) (tablet-download-pool-2886dfbec1e141198b0cf7a435fac498.queue_time_us, tablet-download-pool-2886dfbec1e141198b0cf7a435fac498.queue_time_us) (tablet-download-pool-2886dfbec1e141198b0cf7a435fac498.run_cpu_time_us, tablet-download-pool-2886dfbec1e141198b0cf7a435fac498.run_cpu_time_us) (tablet-download-pool-2886dfbec1e141198b0cf7a435fac498.run_wall_time_us, tablet-download-pool-2886dfbec1e141198b0cf7a435fac498.run_wall_time_us) (tablet-download-pool-387bdd8ae3b34f588e2170b061a5bc05.queue_time_us, tablet-download-pool-387bdd8ae3b34f588e2170b061a5bc05.queue_time_us) (tablet-download-pool-387bdd8ae3b34f588e2170b061a5bc05.run_cpu_time_us, tablet-download-pool-387bdd8ae3b34f588e2170b061a5bc05.run_cpu_time_us) (tablet-download-pool-387bdd8ae3b34f588e2170b061a5bc05.run_wall_time_us, tablet-download-pool-387bdd8ae3b34f588e2170b061a5bc05.run_wall_time_us) (tablet-download-pool-436c127af87c49e8b7bf9b6571f7dc30.queue_time_us, tablet-download-pool-436c127af87c49e8b7bf9b6571f7dc30.queue_time_us) (tablet-download-pool-76f07464d9bd49a9b1b9d58f6a180cf8.queue_time_us, tablet-download-pool-76f07464d9bd49a9b1b9d58f6a180cf8.queue_time_us) (tablet-download-pool-76f07464d9bd49a9b1b9d58f6a180cf8.run_cpu_time_us, tablet-download-pool-76f07464d9bd49a9b1b9d58f6a180cf8.run_cpu_time_us) (tablet-download-pool-76f07464d9bd49a9b1b9d58f6a180cf8.run_wall_time_us, tablet-download-pool-76f07464d9bd49a9b1b9d58f6a180cf8.run_wall_time_us) (tablet-download-pool-7c57cab68b764db0917ed98e74757ff0.queue_time_us, tablet-download-pool-7c57cab68b764db0917ed98e74757ff0.queue_time_us) (tablet-download-pool-7c57cab68b764db0917ed98e74757ff0.run_cpu_time_us, tablet-download-pool-7c57cab68b764db0917ed98e74757ff0.run_cpu_time_us) (tablet-download-pool-7c57cab68b764db0917ed98e74757ff0.run_wall_time_us, tablet-download-pool-7c57cab68b764db0917ed98e74757ff0.run_wall_time_us) (tablet-download-pool-837e8b1235a44131bdaef24b763d1cac.queue_time_us, tablet-download-pool-837e8b1235a44131bdaef24b763d1cac.queue_time_us) (tablet-download-pool-837e8b1235a44131bdaef24b763d1cac.run_cpu_time_us, tablet-download-pool-837e8b1235a44131bdaef24b763d1cac.run_cpu_time_us) (tablet-download-pool-837e8b1235a44131bdaef24b763d1cac.run_wall_time_us, tablet-download-pool-837e8b1235a44131bdaef24b763d1cac.run_wall_time_us) (tablet-download-pool-b6d365c10c76416a944ec8fe409e3dc5.queue_time_us, tablet-download-pool-b6d365c10c76416a944ec8fe409e3dc5.queue_time_us) (tablet-download-pool-b6d365c10c76416a944ec8fe409e3dc5.run_cpu_time_us, tablet-download-pool-b6d365c10c76416a944ec8fe409e3dc5.run_cpu_time_us) (tablet-download-pool-b6d365c10c76416a944ec8fe409e3dc5.run_wall_time_us, tablet-download-pool-b6d365c10c76416a944ec8fe409e3dc5.run_wall_time_us) (tablet-download-pool-bb9f100600a246708457280d91826fce.queue_time_us, tablet-download-pool-bb9f100600a246708457280d91826fce.queue_time_us) (tablet-download-pool-bb9f100600a246708457280d91826fce.run_cpu_time_us, tablet-download-pool-bb9f100600a246708457280d91826fce.run_cpu_time_us) (tablet-download-pool-bb9f100600a246708457280d91826fce.run_wall_time_us, tablet-download-pool-bb9f100600a246708457280d91826fce.run_wall_time_us) (tablet-download-pool-d81f14b7c1d1468196d0fe6d67fe93a7.queue_time_us, tablet-download-pool-d81f14b7c1d1468196d0fe6d67fe93a7.queue_time_us) (tablet-download-pool-d81f14b7c1d1468196d0fe6d67fe93a7.run_cpu_time_us, tablet-download-pool-d81f14b7c1d1468196d0fe6d67fe93a7.run_cpu_time_us) (tablet-download-pool-d81f14b7c1d1468196d0fe6d67fe93a7.run_wall_time_us, tablet-download-pool-d81f14b7c1d1468196d0fe6d67fe93a7.run_wall_time_us) (tablet-download-pool-e48dda3e62e14e46b03d79648e3e2773.queue_time_us, tablet-download-pool-e48dda3e62e14e46b03d79648e3e2773.queue_time_us) (tablet-download-pool-e48dda3e62e14e46b03d79648e3e2773.run_cpu_time_us, tablet-download-pool-e48dda3e62e14e46b03d79648e3e2773.run_cpu_time_us) (tablet-download-pool-e48dda3e62e14e46b03d79648e3e2773.run_wall_time_us, tablet-download-pool-e48dda3e62e14e46b03d79648e3e2773.run_wall_time_us) (tablet-download-pool-e51db70f5f284c4da3c1669fddb41b00.queue_time_us, tablet-download-pool-e51db70f5f284c4da3c1669fddb41b00.queue_time_us) (tablet-download-pool-e51db70f5f284c4da3c1669fddb41b00.run_cpu_time_us, tablet-download-pool-e51db70f5f284c4da3c1669fddb41b00.run_cpu_time_us) (tablet-download-pool-e51db70f5f284c4da3c1669fddb41b00.run_wall_time_us, tablet-download-pool-e51db70f5f284c4da3c1669fddb41b00.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 14:00:39.367483 15964 consensus_queue.cc:237] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:39.369544 17514 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:39.369709 19743 tablet_copy_client.cc:523] T a7981c3f46d8483caae82fbf2296e762 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:39.369973 19218 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.370371 17248 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:39.372016 19743 tablet_bootstrap.cc:492] T a7981c3f46d8483caae82fbf2296e762 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap starting.
I20240502 14:00:39.372256 19672 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:39.372184 15569 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.372545 19672 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:39.374133 17646 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.375219 15702 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:39.376068 19285 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:39.376353 15422 catalog_manager.cc:5484] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 7 to 8, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.376852 17248 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2254b3ee3db647598d781e467584f78d (127.8.86.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.379484 19671 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:39.380256 15964 consensus_queue.cc:237] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } }
I20240502 14:00:39.381582 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e76bd0ecd3e4c168e141fb7984af3ec with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:39.382083 17646 raft_consensus.cc:1261] T 837e8b1235a44131bdaef24b763d1cac P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:39.382633 19285 consensus_queue.cc:1035] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:39.383044 17250 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:39.383394 15429 catalog_manager.cc:5484] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:39.384797 19285 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.385634 17514 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:39.386129 15569 raft_consensus.cc:1261] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:39.386586 17644 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P 2254b3ee3db647598d781e467584f78d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.387990 18566 consensus_queue.cc:1035] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:39.388531 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 837e8b1235a44131bdaef24b763d1cac with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:39.388698 15429 catalog_manager.cc:5484] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 8 to 9, VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:39.389570 15702 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } }
I20240502 14:00:39.390097 15569 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } }
I20240502 14:00:39.393126 17231 tablet_service.cc:1514] Processing DeleteTablet for tablet 837e8b1235a44131bdaef24b763d1cac with delete_type TABLET_DATA_TOMBSTONED (TS f688370715454d6e814a78fcf85ac5c3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:60394
I20240502 14:00:39.393404 19717 tablet_replica.cc:330] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3: stopping tablet replica
I20240502 14:00:39.393783 19717 raft_consensus.cc:2229] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Raft consensus shutting down.
@ 0x7febffad3dcd google::LogMessage::Fail() at ??:0
I20240502 14:00:39.394088 19717 raft_consensus.cc:2258] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:39.395925 19717 ts_tablet_manager.cc:1875] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:39.396329 15549 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e76bd0ecd3e4c168e141fb7984af3ec with delete_type TABLET_DATA_TOMBSTONED (TS 1eb0e3a6c782469fa8110f6bb6a585cf not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:44898
I20240502 14:00:39.396513 19665 tablet_replica.cc:330] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: stopping tablet replica
I20240502 14:00:39.396631 19665 raft_consensus.cc:2229] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:39.396788 19665 raft_consensus.cc:2258] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:39.397455 19665 ts_tablet_manager.cc:1875] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:39.398144 19717 ts_tablet_manager.cc:1888] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:00:39.398218 19717 log.cc:1196] T 837e8b1235a44131bdaef24b763d1cac P f688370715454d6e814a78fcf85ac5c3: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-4/wal/wals/837e8b1235a44131bdaef24b763d1cac
I20240502 14:00:39.399516 19665 ts_tablet_manager.cc:1888] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 14:00:39.399576 19665 log.cc:1196] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/wals/6e76bd0ecd3e4c168e141fb7984af3ec
I20240502 14:00:39.400220 19993 ts_tablet_manager.cc:902] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:39.401034 19993 tablet_copy_client.cc:308] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
@ 0x7febffad7b93 google::LogMessage::SendToLog() at ??:0
I20240502 14:00:39.402120 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 22eef55211ca4f65979022d2f4ae829f from peer 8484181c4ece483c8b408817fab1a917 ({username='slave'} at 127.8.86.130:49307)
I20240502 14:00:39.402222 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 22eef55211ca4f65979022d2f4ae829f from peer 8484181c4ece483c8b408817fab1a917 at {username='slave'} at 127.8.86.130:49307: session id = 8484181c4ece483c8b408817fab1a917-22eef55211ca4f65979022d2f4ae829f
I20240502 14:00:39.403209 15853 tablet_copy_source_session.cc:205] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:39.403560 15414 catalog_manager.cc:4830] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): tablet 837e8b1235a44131bdaef24b763d1cac (table rebalance_test_table_6 [id=6dcd44a470ab434a988eea52c3916f26]) successfully deleted
@ 0x7febffad37cc google::LogMessage::Flush() at ??:0
I20240502 14:00:39.405858 19993 ts_tablet_manager.cc:1875] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:00:39.408385 19993 ts_tablet_manager.cc:1888] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240502 14:00:39.408553 19993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22eef55211ca4f65979022d2f4ae829f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.409761 19993 tablet_copy_client.cc:783] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:39.409901 19993 tablet_copy_client.cc:647] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: tablet copy: Starting download of 1 WAL segments...
@ 0x7febffad4f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 14:00:39.415650 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:39.416472 15415 catalog_manager.cc:4830] TS 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237): tablet 6e76bd0ecd3e4c168e141fb7984af3ec (table rebalance_test_table_4 [id=278d2359eb234a2b8e569cd993a004a7]) successfully deleted
I20240502 14:00:39.420562 19993 tablet_copy_client.cc:523] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:39.421928 15569 consensus_queue.cc:237] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:00:39.423341 17381 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:00:39.424165 17248 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:00:39.424577 18653 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:00:39.424999 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:39.425937 18653 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 14:00:39.427798 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): Couldn't send request to peer 8484181c4ece483c8b408817fab1a917. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:39.431521 19993 tablet_bootstrap.cc:492] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:39.434167 19048 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } } }
I20240502 14:00:39.434667 17248 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } } }
I20240502 14:00:39.437388 15425 catalog_manager.cc:5484] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 11 to 12, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:39.438169 17381 raft_consensus.cc:2932] T dfdcf1124e8a474facc14c45e8963568 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } } }
I20240502 14:00:39.439903 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7fec00f3d044 kudu::TraceMetrics::InternName() at ??:0
I20240502 14:00:39.446956 19903 tablet_bootstrap.cc:492] T 387bdd8ae3b34f588e2170b061a5bc05 P 2254b3ee3db647598d781e467584f78d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:39.448698 19903 tablet_bootstrap.cc:492] T 387bdd8ae3b34f588e2170b061a5bc05 P 2254b3ee3db647598d781e467584f78d: Bootstrap complete.
I20240502 14:00:39.448913 19903 ts_tablet_manager.cc:1367] T 387bdd8ae3b34f588e2170b061a5bc05 P 2254b3ee3db647598d781e467584f78d: Time spent bootstrapping tablet: real 0.181s user 0.030s sys 0.004s
I20240502 14:00:39.449388 19903 raft_consensus.cc:348] T 387bdd8ae3b34f588e2170b061a5bc05 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:39.449523 19903 raft_consensus.cc:729] T 387bdd8ae3b34f588e2170b061a5bc05 P 2254b3ee3db647598d781e467584f78d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2254b3ee3db647598d781e467584f78d, State: Initialized, Role: LEARNER
I20240502 14:00:39.449654 19903 consensus_queue.cc:260] T 387bdd8ae3b34f588e2170b061a5bc05 P 2254b3ee3db647598d781e467584f78d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:39.450019 19903 ts_tablet_manager.cc:1398] T 387bdd8ae3b34f588e2170b061a5bc05 P 2254b3ee3db647598d781e467584f78d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
@ 0x7fec00f21f98 kudu::ThreadPool::ThreadPool() at ??:0
I20240502 14:00:39.452193 15984 tablet_copy_service.cc:342] P 4d0f8f311e834d81bd54b9f8e7923bcd: Request end of tablet copy session 2254b3ee3db647598d781e467584f78d-387bdd8ae3b34f588e2170b061a5bc05 received from {username='slave'} at 127.8.86.136:55499
I20240502 14:00:39.452275 15984 tablet_copy_service.cc:433] P 4d0f8f311e834d81bd54b9f8e7923bcd: ending tablet copy session 2254b3ee3db647598d781e467584f78d-387bdd8ae3b34f588e2170b061a5bc05 on tablet 387bdd8ae3b34f588e2170b061a5bc05 with peer 2254b3ee3db647598d781e467584f78d
I20240502 14:00:39.454738 20004 ts_tablet_manager.cc:902] T 2d92e5885bf94c3e8e0e4dee40017699 P 3c3658c946994e2ba9f41b64aa327d25: Initiating tablet copy from peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:00:39.455318 20004 tablet_copy_client.cc:308] T 2d92e5885bf94c3e8e0e4dee40017699 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.132:45719
I20240502 14:00:39.455749 15984 tablet_copy_service.cc:140] P 4d0f8f311e834d81bd54b9f8e7923bcd: Received BeginTabletCopySession request for tablet 2d92e5885bf94c3e8e0e4dee40017699 from peer 3c3658c946994e2ba9f41b64aa327d25 ({username='slave'} at 127.8.86.134:37349)
I20240502 14:00:39.455855 15984 tablet_copy_service.cc:161] P 4d0f8f311e834d81bd54b9f8e7923bcd: Beginning new tablet copy session on tablet 2d92e5885bf94c3e8e0e4dee40017699 from peer 3c3658c946994e2ba9f41b64aa327d25 at {username='slave'} at 127.8.86.134:37349: session id = 3c3658c946994e2ba9f41b64aa327d25-2d92e5885bf94c3e8e0e4dee40017699
I20240502 14:00:39.456947 15984 tablet_copy_source_session.cc:205] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7fec00f1fefc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fec03ad277a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 14:00:39.462808 20004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d92e5885bf94c3e8e0e4dee40017699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.465222 20004 tablet_copy_client.cc:783] T 2d92e5885bf94c3e8e0e4dee40017699 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:39.465360 20004 tablet_copy_client.cc:647] T 2d92e5885bf94c3e8e0e4dee40017699 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:39.478132 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:39.481962 15569 consensus_queue.cc:237] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
@ 0x7fec03adba45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 14:00:39.486011 17381 raft_consensus.cc:1261] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 14:00:39.487016 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0cd21dc897b41aca9118be6df48cc0c. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:39.487172 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0cd21dc897b41aca9118be6df48cc0c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:39.487363 19694 consensus_queue.cc:1035] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:39.488082 15964 raft_consensus.cc:1261] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:39.488469 19694 consensus_queue.cc:1035] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:39.490348 20004 tablet_copy_client.cc:523] T 2d92e5885bf94c3e8e0e4dee40017699 P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:39.490943 19694 raft_consensus.cc:2932] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:39.492094 20004 tablet_bootstrap.cc:492] T 2d92e5885bf94c3e8e0e4dee40017699 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap starting.
I20240502 14:00:39.493594 17381 raft_consensus.cc:2932] T b0cd21dc897b41aca9118be6df48cc0c P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:39.494102 15425 catalog_manager.cc:5484] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 7 to 8, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:39.500288 15964 raft_consensus.cc:2932] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:39.501627 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
@ 0x7fec03b5c76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 14:00:39.514076 20052 ts_tablet_manager.cc:902] T 1f80c8590e994a10a9b7c6c174e28173 P f688370715454d6e814a78fcf85ac5c3: Initiating tablet copy from peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:39.514650 20052 tablet_copy_client.cc:308] T 1f80c8590e994a10a9b7c6c174e28173 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:42261
I20240502 14:00:39.515522 15721 tablet_copy_service.cc:140] P 8484181c4ece483c8b408817fab1a917: Received BeginTabletCopySession request for tablet 1f80c8590e994a10a9b7c6c174e28173 from peer f688370715454d6e814a78fcf85ac5c3 ({username='slave'} at 127.8.86.133:54505)
I20240502 14:00:39.515619 15721 tablet_copy_service.cc:161] P 8484181c4ece483c8b408817fab1a917: Beginning new tablet copy session on tablet 1f80c8590e994a10a9b7c6c174e28173 from peer f688370715454d6e814a78fcf85ac5c3 at {username='slave'} at 127.8.86.133:54505: session id = f688370715454d6e814a78fcf85ac5c3-1f80c8590e994a10a9b7c6c174e28173
I20240502 14:00:39.515880 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:39.516618 15721 tablet_copy_source_session.cc:205] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:39.521778 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
@ 0x7fec03b704f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 14:00:39.536486 17627 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7fec03b5b226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240502 14:00:39.537901 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:39.543040 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7fec03b6ac0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x555c5d97575c std::function<>::operator()() at ??:0
I20240502 14:00:39.554193 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:39.560753 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:39.570003 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
@ 0x7fec00f24e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fec00f256ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fec00f2703b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x555c5d97575c std::function<>::operator()() at ??:0
@ 0x7fec00f14807 kudu::Thread::SuperviseThread() at ??:0
I20240502 14:00:39.596035 20052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f80c8590e994a10a9b7c6c174e28173. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fec017526db start_thread at ??:0
@ 0x7febfec8a71f clone at ??:0
I20240502 14:00:39.607510 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:39.613082 19919 ts_tablet_manager.cc:902] T 2886dfbec1e141198b0cf7a435fac498 P 8fea283177394bd28d7eacf4fb744d59: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:39.613639 19919 tablet_copy_client.cc:308] T 2886dfbec1e141198b0cf7a435fac498 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
I20240502 14:00:39.614197 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 2886dfbec1e141198b0cf7a435fac498 from peer 8fea283177394bd28d7eacf4fb744d59 ({username='slave'} at 127.8.86.135:39871)
I20240502 14:00:39.614341 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 2886dfbec1e141198b0cf7a435fac498 from peer 8fea283177394bd28d7eacf4fb744d59 at {username='slave'} at 127.8.86.135:39871: session id = 8fea283177394bd28d7eacf4fb744d59-2886dfbec1e141198b0cf7a435fac498
I20240502 14:00:39.615334 15853 tablet_copy_source_session.cc:205] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:39.619167 19919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2886dfbec1e141198b0cf7a435fac498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.621659 19919 tablet_copy_client.cc:783] T 2886dfbec1e141198b0cf7a435fac498 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:39.621814 19919 tablet_copy_client.cc:647] T 2886dfbec1e141198b0cf7a435fac498 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:39.626463 20052 tablet_copy_client.cc:783] T 1f80c8590e994a10a9b7c6c174e28173 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:39.626703 20052 tablet_copy_client.cc:647] T 1f80c8590e994a10a9b7c6c174e28173 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:39.628373 20052 tablet_copy_client.cc:523] T 1f80c8590e994a10a9b7c6c174e28173 P f688370715454d6e814a78fcf85ac5c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:39.629832 20052 tablet_bootstrap.cc:492] T 1f80c8590e994a10a9b7c6c174e28173 P f688370715454d6e814a78fcf85ac5c3: Bootstrap starting.
I20240502 14:00:39.640152 19993 tablet_bootstrap.cc:492] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:39.640564 19993 tablet_bootstrap.cc:492] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: Bootstrap complete.
I20240502 14:00:39.640930 19993 ts_tablet_manager.cc:1367] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.209s user 0.031s sys 0.002s
I20240502 14:00:39.641541 19993 raft_consensus.cc:348] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:00:39.641681 19993 raft_consensus.cc:729] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: LEARNER
I20240502 14:00:39.641786 19993 consensus_queue.cc:260] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:00:39.642081 19993 ts_tablet_manager.cc:1398] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:39.642630 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session 8484181c4ece483c8b408817fab1a917-22eef55211ca4f65979022d2f4ae829f received from {username='slave'} at 127.8.86.130:49307
I20240502 14:00:39.642707 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session 8484181c4ece483c8b408817fab1a917-22eef55211ca4f65979022d2f4ae829f on tablet 22eef55211ca4f65979022d2f4ae829f with peer 8484181c4ece483c8b408817fab1a917
I20240502 14:00:39.643117 15569 consensus_queue.cc:237] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } }
I20240502 14:00:39.644680 15832 raft_consensus.cc:1261] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:39.645077 19694 consensus_queue.cc:1035] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 14:00:39.645323 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb9f100600a246708457280d91826fce. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:39.645421 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719): Couldn't send request to peer 4d0f8f311e834d81bd54b9f8e7923bcd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb9f100600a246708457280d91826fce. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:39.646883 18653 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } } }
I20240502 14:00:39.650250 15832 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } } }
I20240502 14:00:39.653399 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:39.654243 19743 tablet_bootstrap.cc:492] T a7981c3f46d8483caae82fbf2296e762 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:39.654098 15425 catalog_manager.cc:5484] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 7 to 8, NON_VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } } }
I20240502 14:00:39.654618 19919 tablet_copy_client.cc:523] T 2886dfbec1e141198b0cf7a435fac498 P 8fea283177394bd28d7eacf4fb744d59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:39.656174 19919 tablet_bootstrap.cc:492] T 2886dfbec1e141198b0cf7a435fac498 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap starting.
I20240502 14:00:39.659493 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:39.670449 19743 tablet_bootstrap.cc:492] T a7981c3f46d8483caae82fbf2296e762 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap complete.
I20240502 14:00:39.670722 19743 ts_tablet_manager.cc:1367] T a7981c3f46d8483caae82fbf2296e762 P 3c3658c946994e2ba9f41b64aa327d25: Time spent bootstrapping tablet: real 0.299s user 0.029s sys 0.001s
I20240502 14:00:39.671185 19743 raft_consensus.cc:348] T a7981c3f46d8483caae82fbf2296e762 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:39.671300 19743 raft_consensus.cc:729] T a7981c3f46d8483caae82fbf2296e762 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c3658c946994e2ba9f41b64aa327d25, State: Initialized, Role: LEARNER
I20240502 14:00:39.671392 19743 consensus_queue.cc:260] T a7981c3f46d8483caae82fbf2296e762 P 3c3658c946994e2ba9f41b64aa327d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:39.672266 19743 ts_tablet_manager.cc:1398] T a7981c3f46d8483caae82fbf2296e762 P 3c3658c946994e2ba9f41b64aa327d25: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:39.672778 15721 tablet_copy_service.cc:342] P 8484181c4ece483c8b408817fab1a917: Request end of tablet copy session 3c3658c946994e2ba9f41b64aa327d25-a7981c3f46d8483caae82fbf2296e762 received from {username='slave'} at 127.8.86.134:44409
I20240502 14:00:39.672863 15721 tablet_copy_service.cc:433] P 8484181c4ece483c8b408817fab1a917: ending tablet copy session 3c3658c946994e2ba9f41b64aa327d25-a7981c3f46d8483caae82fbf2296e762 on tablet a7981c3f46d8483caae82fbf2296e762 with peer 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:39.690084 20087 ts_tablet_manager.cc:902] T bb9f100600a246708457280d91826fce P 3c3658c946994e2ba9f41b64aa327d25: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:39.693248 20087 tablet_copy_client.cc:308] T bb9f100600a246708457280d91826fce P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:39.693756 15591 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet bb9f100600a246708457280d91826fce from peer 3c3658c946994e2ba9f41b64aa327d25 ({username='slave'} at 127.8.86.134:43305)
I20240502 14:00:39.693863 15591 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet bb9f100600a246708457280d91826fce from peer 3c3658c946994e2ba9f41b64aa327d25 at {username='slave'} at 127.8.86.134:43305: session id = 3c3658c946994e2ba9f41b64aa327d25-bb9f100600a246708457280d91826fce
I20240502 14:00:39.694813 15591 tablet_copy_source_session.cc:205] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:39.695761 20087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9f100600a246708457280d91826fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.705227 19919 tablet_bootstrap.cc:492] T 2886dfbec1e141198b0cf7a435fac498 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:39.705525 19919 tablet_bootstrap.cc:492] T 2886dfbec1e141198b0cf7a435fac498 P 8fea283177394bd28d7eacf4fb744d59: Bootstrap complete.
I20240502 14:00:39.710343 19919 ts_tablet_manager.cc:1367] T 2886dfbec1e141198b0cf7a435fac498 P 8fea283177394bd28d7eacf4fb744d59: Time spent bootstrapping tablet: real 0.054s user 0.026s sys 0.005s
I20240502 14:00:39.710862 19919 raft_consensus.cc:348] T 2886dfbec1e141198b0cf7a435fac498 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:39.710981 19919 raft_consensus.cc:729] T 2886dfbec1e141198b0cf7a435fac498 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8fea283177394bd28d7eacf4fb744d59, State: Initialized, Role: LEARNER
I20240502 14:00:39.711082 19919 consensus_queue.cc:260] T 2886dfbec1e141198b0cf7a435fac498 P 8fea283177394bd28d7eacf4fb744d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:00:39.711447 19919 ts_tablet_manager.cc:1398] T 2886dfbec1e141198b0cf7a435fac498 P 8fea283177394bd28d7eacf4fb744d59: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:39.712183 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session 8fea283177394bd28d7eacf4fb744d59-2886dfbec1e141198b0cf7a435fac498 received from {username='slave'} at 127.8.86.135:39871
I20240502 14:00:39.712266 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session 8fea283177394bd28d7eacf4fb744d59-2886dfbec1e141198b0cf7a435fac498 on tablet 2886dfbec1e141198b0cf7a435fac498 with peer 8fea283177394bd28d7eacf4fb744d59
I20240502 14:00:39.728781 20052 tablet_bootstrap.cc:492] T 1f80c8590e994a10a9b7c6c174e28173 P f688370715454d6e814a78fcf85ac5c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:39.729081 20052 tablet_bootstrap.cc:492] T 1f80c8590e994a10a9b7c6c174e28173 P f688370715454d6e814a78fcf85ac5c3: Bootstrap complete.
I20240502 14:00:39.729274 20052 ts_tablet_manager.cc:1367] T 1f80c8590e994a10a9b7c6c174e28173 P f688370715454d6e814a78fcf85ac5c3: Time spent bootstrapping tablet: real 0.099s user 0.028s sys 0.000s
I20240502 14:00:39.728668 20087 tablet_copy_client.cc:783] T bb9f100600a246708457280d91826fce P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:39.729527 20087 tablet_copy_client.cc:647] T bb9f100600a246708457280d91826fce P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:39.730098 20052 raft_consensus.cc:348] T 1f80c8590e994a10a9b7c6c174e28173 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:39.730235 20052 raft_consensus.cc:729] T 1f80c8590e994a10a9b7c6c174e28173 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f688370715454d6e814a78fcf85ac5c3, State: Initialized, Role: LEARNER
I20240502 14:00:39.730365 20052 consensus_queue.cc:260] T 1f80c8590e994a10a9b7c6c174e28173 P f688370715454d6e814a78fcf85ac5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:39.730680 20052 ts_tablet_manager.cc:1398] T 1f80c8590e994a10a9b7c6c174e28173 P f688370715454d6e814a78fcf85ac5c3: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 14:00:39.731353 15721 tablet_copy_service.cc:342] P 8484181c4ece483c8b408817fab1a917: Request end of tablet copy session f688370715454d6e814a78fcf85ac5c3-1f80c8590e994a10a9b7c6c174e28173 received from {username='slave'} at 127.8.86.133:54505
I20240502 14:00:39.731439 15721 tablet_copy_service.cc:433] P 8484181c4ece483c8b408817fab1a917: ending tablet copy session f688370715454d6e814a78fcf85ac5c3-1f80c8590e994a10a9b7c6c174e28173 on tablet 1f80c8590e994a10a9b7c6c174e28173 with peer f688370715454d6e814a78fcf85ac5c3
I20240502 14:00:39.732518 20087 tablet_copy_client.cc:523] T bb9f100600a246708457280d91826fce P 3c3658c946994e2ba9f41b64aa327d25: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:39.743266 19743 ts_tablet_manager.cc:902] T 28e371c1a999431499e09479e8f5b44b P 3c3658c946994e2ba9f41b64aa327d25: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
F20240502 14:00:39.743427 19743 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-03dd735113ab4ea6bdb696b11a4efd7f.queue_time_us, tablet-download-pool-03dd735113ab4ea6bdb696b11a4efd7f.queue_time_us) (tablet-download-pool-03dd735113ab4ea6bdb696b11a4efd7f.run_cpu_time_us, tablet-download-pool-03dd735113ab4ea6bdb696b11a4efd7f.run_cpu_time_us) (tablet-download-pool-03dd735113ab4ea6bdb696b11a4efd7f.run_wall_time_us, tablet-download-pool-03dd735113ab4ea6bdb696b11a4efd7f.run_wall_time_us) (tablet-download-pool-186a946321e445b58b0097c658847ebe.queue_time_us, tablet-download-pool-186a946321e445b58b0097c658847ebe.queue_time_us) (tablet-download-pool-186a946321e445b58b0097c658847ebe.run_cpu_time_us, tablet-download-pool-186a946321e445b58b0097c658847ebe.run_cpu_time_us) (tablet-download-pool-186a946321e445b58b0097c658847ebe.run_wall_time_us, tablet-download-pool-186a946321e445b58b0097c658847ebe.run_wall_time_us) (tablet-download-pool-1f80c8590e994a10a9b7c6c174e28173.queue_time_us, tablet-download-pool-1f80c8590e994a10a9b7c6c174e28173.queue_time_us) (tablet-download-pool-1f80c8590e994a10a9b7c6c174e28173.run_cpu_time_us, tablet-download-pool-1f80c8590e994a10a9b7c6c174e28173.run_cpu_time_us) (tablet-download-pool-1f80c8590e994a10a9b7c6c174e28173.run_wall_time_us, tablet-download-pool-1f80c8590e994a10a9b7c6c174e28173.run_wall_time_us) (tablet-download-pool-28e371c1a999431499e09479e8f5b44b.queue_time_us, tablet-download-pool-28e371c1a999431499e09479e8f5b44b.queue_time_us) (tablet-download-pool-2d92e5885bf94c3e8e0e4dee40017699.queue_time_us, tablet-download-pool-2d92e5885bf94c3e8e0e4dee40017699.queue_time_us) (tablet-download-pool-2d92e5885bf94c3e8e0e4dee40017699.run_cpu_time_us, tablet-download-pool-2d92e5885bf94c3e8e0e4dee40017699.run_cpu_time_us) (tablet-download-pool-2d92e5885bf94c3e8e0e4dee40017699.run_wall_time_us, tablet-download-pool-2d92e5885bf94c3e8e0e4dee40017699.run_wall_time_us) (tablet-download-pool-387bdd8ae3b34f588e2170b061a5bc05.queue_time_us, tablet-download-pool-387bdd8ae3b34f588e2170b061a5bc05.queue_time_us) (tablet-download-pool-387bdd8ae3b34f588e2170b061a5bc05.run_cpu_time_us, tablet-download-pool-387bdd8ae3b34f588e2170b061a5bc05.run_cpu_time_us) (tablet-download-pool-387bdd8ae3b34f588e2170b061a5bc05.run_wall_time_us, tablet-download-pool-387bdd8ae3b34f588e2170b061a5bc05.run_wall_time_us) (tablet-download-pool-436c127af87c49e8b7bf9b6571f7dc30.queue_time_us, tablet-download-pool-436c127af87c49e8b7bf9b6571f7dc30.queue_time_us) (tablet-download-pool-436c127af87c49e8b7bf9b6571f7dc30.run_cpu_time_us, tablet-download-pool-436c127af87c49e8b7bf9b6571f7dc30.run_cpu_time_us) (tablet-download-pool-436c127af87c49e8b7bf9b6571f7dc30.run_wall_time_us, tablet-download-pool-436c127af87c49e8b7bf9b6571f7dc30.run_wall_time_us) (tablet-download-pool-619a8ec66ba74ecea689c17b4734253a.queue_time_us, tablet-download-pool-619a8ec66ba74ecea689c17b4734253a.queue_time_us) (tablet-download-pool-619a8ec66ba74ecea689c17b4734253a.run_cpu_time_us, tablet-download-pool-619a8ec66ba74ecea689c17b4734253a.run_cpu_time_us) (tablet-download-pool-619a8ec66ba74ecea689c17b4734253a.run_wall_time_us, tablet-download-pool-619a8ec66ba74ecea689c17b4734253a.run_wall_time_us) (tablet-download-pool-76f07464d9bd49a9b1b9d58f6a180cf8.queue_time_us, tablet-download-pool-76f07464d9bd49a9b1b9d58f6a180cf8.queue_time_us) (tablet-download-pool-76f07464d9bd49a9b1b9d58f6a180cf8.run_cpu_time_us, tablet-download-pool-76f07464d9bd49a9b1b9d58f6a180cf8.run_cpu_time_us) (tablet-download-pool-76f07464d9bd49a9b1b9d58f6a180cf8.run_wall_time_us, tablet-download-pool-76f07464d9bd49a9b1b9d58f6a180cf8.run_wall_time_us) (tablet-download-pool-9391f16ae2d247ec8a71e5c971cdda71.queue_time_us, tablet-download-pool-9391f16ae2d247ec8a71e5c971cdda71.queue_time_us) (tablet-download-pool-9391f16ae2d247ec8a71e5c971cdda71.run_cpu_time_us, tablet-download-pool-9391f16ae2d247ec8a71e5c971cdda71.run_cpu_time_us) (tablet-download-pool-9391f16ae2d247ec8a71e5c971cdda71.run_wall_time_us, tablet-download-pool-9391f16ae2d247ec8a71e5c971cdda71.run_wall_time_us) (tablet-download-pool-9870bc797f184a6284ff6ca4f78b3826.queue_time_us, tablet-download-pool-9870bc797f184a6284ff6ca4f78b3826.queue_time_us) (tablet-download-pool-9870bc797f184a6284ff6ca4f78b3826.run_cpu_time_us, tablet-download-pool-9870bc797f184a6284ff6ca4f78b3826.run_cpu_time_us) (tablet-download-pool-9870bc797f184a6284ff6ca4f78b3826.run_wall_time_us, tablet-download-pool-9870bc797f184a6284ff6ca4f78b3826.run_wall_time_us) (tablet-download-pool-a7981c3f46d8483caae82fbf2296e762.queue_time_us, tablet-download-pool-a7981c3f46d8483caae82fbf2296e762.queue_time_us) (tablet-download-pool-a7981c3f46d8483caae82fbf2296e762.run_cpu_time_us, tablet-download-pool-a7981c3f46d8483caae82fbf2296e762.run_cpu_time_us) (tablet-download-pool-a7981c3f46d8483caae82fbf2296e762.run_wall_time_us, tablet-download-pool-a7981c3f46d8483caae82fbf2296e762.run_wall_time_us) (tablet-download-pool-b0cd21dc897b41aca9118be6df48cc0c.queue_time_us, tablet-download-pool-b0cd21dc897b41aca9118be6df48cc0c.queue_time_us) (tablet-download-pool-b0cd21dc897b41aca9118be6df48cc0c.run_cpu_time_us, tablet-download-pool-b0cd21dc897b41aca9118be6df48cc0c.run_cpu_time_us) (tablet-download-pool-b0cd21dc897b41aca9118be6df48cc0c.run_wall_time_us, tablet-download-pool-b0cd21dc897b41aca9118be6df48cc0c.run_wall_time_us) (tablet-download-pool-bb9f100600a246708457280d91826fce.queue_time_us, tablet-download-pool-bb9f100600a246708457280d91826fce.queue_time_us) (tablet-download-pool-bb9f100600a246708457280d91826fce.run_cpu_time_us, tablet-download-pool-bb9f100600a246708457280d91826fce.run_cpu_time_us) (tablet-download-pool-bb9f100600a246708457280d91826fce.run_wall_time_us, tablet-download-pool-bb9f100600a246708457280d91826fce.run_wall_time_us) (tablet-download-pool-dfdcf1124e8a474facc14c45e8963568.queue_time_us, tablet-download-pool-dfdcf1124e8a474facc14c45e8963568.queue_time_us) (tablet-download-pool-dfdcf1124e8a474facc14c45e8963568.run_cpu_time_us, tablet-download-pool-dfdcf1124e8a474facc14c45e8963568.run_cpu_time_us) (tablet-download-pool-dfdcf1124e8a474facc14c45e8963568.run_wall_time_us, tablet-download-pool-dfdcf1124e8a474facc14c45e8963568.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 14:00:39.743767 15702 consensus_queue.cc:237] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } }
I20240502 14:00:39.746169 17514 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:39.746618 17381 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 14:00:39.746778 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719): Couldn't send request to peer 4d0f8f311e834d81bd54b9f8e7923bcd. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:39.746986 18605 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:39.747498 18605 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
@ 0x7ff2979cbdcd google::LogMessage::Fail() at ??:0
I20240502 14:00:39.748052 15569 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:39.748656 18605 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
@ 0x7ff2979cfb93 google::LogMessage::SendToLog() at ??:0
@ 0x7ff2979cb7cc google::LogMessage::Flush() at ??:0
I20240502 14:00:39.752758 18605 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } } }
I20240502 14:00:39.755018 19826 raft_consensus.cc:1050] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917: attempting to promote NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 to VOTER
@ 0x7ff2979ccf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 14:00:39.755344 19826 consensus_queue.cc:237] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } }
I20240502 14:00:39.756516 17381 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } } }
I20240502 14:00:39.757233 17514 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } } }
I20240502 14:00:39.757213 15569 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } } }
I20240502 14:00:39.760325 15423 catalog_manager.cc:5484] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 9 to 10, NON_VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } } }
@ 0x7ff298e35044 kudu::TraceMetrics::InternName() at ??:0
I20240502 14:00:39.763808 15569 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:00:39.764287 18174 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 14:00:39.764418 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719): Couldn't send request to peer 4d0f8f311e834d81bd54b9f8e7923bcd. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:39.764794 17514 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:00:39.765058 17381 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:00:39.765578 18174 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:00:39.765694 19794 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
@ 0x7ff298e19f98 kudu::ThreadPool::ThreadPool() at ??:0
I20240502 14:00:39.769156 18174 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } } }
@ 0x7ff298e17efc kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 14:00:39.771340 17380 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } } }
I20240502 14:00:39.772547 17515 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } } }
I20240502 14:00:39.774784 15426 catalog_manager.cc:5484] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 10 to 11, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
@ 0x7ff29b9ca77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 14:00:39.775575 15569 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } } }
I20240502 14:00:39.778136 17514 raft_consensus.cc:1203] T 436c127af87c49e8b7bf9b6571f7dc30 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
@ 0x7ff29b9d3a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 14:00:39.781262 15702 consensus_queue.cc:237] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }
@ 0x7ff29ba5476f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 14:00:39.783262 15567 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:00:39.783890 17380 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:00:39.784106 19826 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:00:39.784379 19826 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:00:39.785130 17514 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:00:39.785526 18605 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
@ 0x7ff29ba684f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 14:00:39.787349 19794 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:39.790352 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a7981c3f46d8483caae82fbf2296e762 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
@ 0x7ff29ba53226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240502 14:00:39.790989 17380 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
@ 0x7ff29ba62c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x564b36a0275c std::function<>::operator()() at ??:0
@ 0x7ff298e1ce06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff298e1d6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240502 14:00:39.836695 17231 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:39.814052 15964 consensus_queue.cc:237] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:39.833343 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:39.830703 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:39.805460 20004 tablet_bootstrap.cc:492] T 2d92e5885bf94c3e8e0e4dee40017699 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:39.792083 15426 catalog_manager.cc:5484] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 11 to 12, NON_VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:39.877404 20004 tablet_bootstrap.cc:492] T 2d92e5885bf94c3e8e0e4dee40017699 P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap complete.
I20240502 14:00:39.877652 20004 ts_tablet_manager.cc:1367] T 2d92e5885bf94c3e8e0e4dee40017699 P 3c3658c946994e2ba9f41b64aa327d25: Time spent bootstrapping tablet: real 0.386s user 0.031s sys 0.000s
I20240502 14:00:39.792848 15569 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:39.878118 20004 raft_consensus.cc:348] T 2d92e5885bf94c3e8e0e4dee40017699 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:39.878238 20004 raft_consensus.cc:729] T 2d92e5885bf94c3e8e0e4dee40017699 P 3c3658c946994e2ba9f41b64aa327d25 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c3658c946994e2ba9f41b64aa327d25, State: Initialized, Role: LEARNER
I20240502 14:00:39.795517 17514 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:39.805915 20087 tablet_bootstrap.cc:492] T bb9f100600a246708457280d91826fce P 3c3658c946994e2ba9f41b64aa327d25: Bootstrap starting.
I20240502 14:00:39.823668 17362 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:39.883989 15944 tablet_service.cc:1514] Processing DeleteTablet for tablet a7981c3f46d8483caae82fbf2296e762 with delete_type TABLET_DATA_TOMBSTONED (TS 4d0f8f311e834d81bd54b9f8e7923bcd not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:56876
I20240502 14:00:39.889362 15413 catalog_manager.cc:4830] TS 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719): tablet a7981c3f46d8483caae82fbf2296e762 (table rebalance_test_table_2 [id=b36f1f16cb3d4b8bae9303e2976dc2e7]) successfully deleted
I20240502 14:00:39.896220 20004 consensus_queue.cc:260] T 2d92e5885bf94c3e8e0e4dee40017699 P 3c3658c946994e2ba9f41b64aa327d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } }
I20240502 14:00:39.896612 20004 ts_tablet_manager.cc:1398] T 2d92e5885bf94c3e8e0e4dee40017699 P 3c3658c946994e2ba9f41b64aa327d25: Time spent starting tablet: real 0.019s user 0.001s sys 0.000s
@ 0x7ff298e1f03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x564b36a0275c std::function<>::operator()() at ??:0
@ 0x7ff298e0c807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff29964a6db start_thread at ??:0
W20240502 14:00:39.907549 15898 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a5e33f337bd47b2a6b11b29a5eacad8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:39.907588 15984 tablet_copy_service.cc:342] P 4d0f8f311e834d81bd54b9f8e7923bcd: Request end of tablet copy session 3c3658c946994e2ba9f41b64aa327d25-2d92e5885bf94c3e8e0e4dee40017699 received from {username='slave'} at 127.8.86.134:37349
I20240502 14:00:39.907665 15984 tablet_copy_service.cc:433] P 4d0f8f311e834d81bd54b9f8e7923bcd: ending tablet copy session 3c3658c946994e2ba9f41b64aa327d25-2d92e5885bf94c3e8e0e4dee40017699 on tablet 2d92e5885bf94c3e8e0e4dee40017699 with peer 3c3658c946994e2ba9f41b64aa327d25
@ 0x7ff296b8271f clone at ??:0
I20240502 14:00:39.909621 15702 consensus_queue.cc:237] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }
I20240502 14:00:39.909879 15944 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:39.910002 15832 raft_consensus.cc:1261] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:39.910338 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:00:39.911409 19285 consensus_queue.cc:1035] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:39.914937 17514 raft_consensus.cc:1261] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:39.915335 19285 consensus_queue.cc:1035] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:39.918588 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:39.919240 20052 ts_tablet_manager.cc:902] T b0cd21dc897b41aca9118be6df48cc0c P f688370715454d6e814a78fcf85ac5c3: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:39.919414 17515 raft_consensus.cc:1261] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 14:00:39.919698 20052 tablet_copy_client.cc:308] T b0cd21dc897b41aca9118be6df48cc0c P f688370715454d6e814a78fcf85ac5c3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:39.920115 19993 ts_tablet_manager.cc:902] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:39.920271 19826 consensus_queue.cc:1035] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:00:39.920558 19993 tablet_copy_client.cc:308] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:39.921466 15591 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet dfdcf1124e8a474facc14c45e8963568 from peer 8484181c4ece483c8b408817fab1a917 ({username='slave'} at 127.8.86.130:49841)
I20240502 14:00:39.921558 15591 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet dfdcf1124e8a474facc14c45e8963568 from peer 8484181c4ece483c8b408817fab1a917 at {username='slave'} at 127.8.86.130:49841: session id = 8484181c4ece483c8b408817fab1a917-dfdcf1124e8a474facc14c45e8963568
I20240502 14:00:39.922449 15591 tablet_copy_source_session.cc:205] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:39.922731 15590 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet b0cd21dc897b41aca9118be6df48cc0c from peer f688370715454d6e814a78fcf85ac5c3 ({username='slave'} at 127.8.86.133:50351)
I20240502 14:00:39.922801 15590 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet b0cd21dc897b41aca9118be6df48cc0c from peer f688370715454d6e814a78fcf85ac5c3 at {username='slave'} at 127.8.86.133:50351: session id = f688370715454d6e814a78fcf85ac5c3-b0cd21dc897b41aca9118be6df48cc0c
I20240502 14:00:39.923523 15590 tablet_copy_source_session.cc:205] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:39.924306 20052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0cd21dc897b41aca9118be6df48cc0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.927951 19993 ts_tablet_manager.cc:1875] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:00:39.930116 19993 ts_tablet_manager.cc:1888] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240502 14:00:39.930281 19993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfdcf1124e8a474facc14c45e8963568. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:00:39.984184 15504 connection.cc:535] client connection to 127.8.86.136:38631 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:00:39.936883 17515 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:39.926990 19285 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
W20240502 14:00:40.005362 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:00:40.006616 15504 consensus_peers.cc:462] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:40.006714 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:40.006754 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
I20240502 14:00:39.968461 17514 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
W20240502 14:00:40.006814 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:39.983633 15897 connection.cc:535] client connection to 127.8.86.136:38631 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:00:40.007042 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:00:40.014176 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:40.033272 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:40.009759 17514 raft_consensus.cc:1203] T 2886dfbec1e141198b0cf7a435fac498 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 14:00:40.009040 15422 catalog_manager.cc:5484] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 6 to 7, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:39.931746 19993 tablet_copy_client.cc:783] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:40.056427 19993 tablet_copy_client.cc:647] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:39.928010 15832 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:00:39.939433 20052 tablet_copy_client.cc:783] T b0cd21dc897b41aca9118be6df48cc0c P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:40.058048 20052 tablet_copy_client.cc:647] T b0cd21dc897b41aca9118be6df48cc0c P f688370715454d6e814a78fcf85ac5c3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:39.935293 19826 raft_consensus.cc:2932] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } }
I20240502 14:00:40.059715 20052 tablet_copy_client.cc:523] T b0cd21dc897b41aca9118be6df48cc0c P f688370715454d6e814a78fcf85ac5c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 14:00:39.984362 15766 connection.cc:535] client connection to 127.8.86.136:38631 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:00:40.060658 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:00:40.061291 20052 tablet_bootstrap.cc:492] T b0cd21dc897b41aca9118be6df48cc0c P f688370715454d6e814a78fcf85ac5c3: Bootstrap starting.
I20240502 14:00:40.021080 17248 raft_consensus.cc:1203] T 1f80c8590e994a10a9b7c6c174e28173 P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:00:39.976886 15702 raft_consensus.cc:1203] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
W20240502 14:00:39.984040 15635 connection.cc:535] client connection to 127.8.86.136:38631 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:00:40.062450 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:00:40.024971 15700 consensus_queue.cc:237] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } }
I20240502 14:00:40.064363 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a7981c3f46d8483caae82fbf2296e762 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:40.064839 15429 catalog_manager.cc:5484] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 12 to 13, VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) evicted. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:40.066161 17248 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 14:00:40.066241 15635 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719): Couldn't send request to peer 4d0f8f311e834d81bd54b9f8e7923bcd. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:40.066905 15831 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:40.067914 19794 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:40.068339 19756 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:40.069926 15549 tablet_service.cc:1514] Processing DeleteTablet for tablet a7981c3f46d8483caae82fbf2296e762 with delete_type TABLET_DATA_TOMBSTONED (TS 1eb0e3a6c782469fa8110f6bb6a585cf not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:44898
W20240502 14:00:40.070606 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:40.070706 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:40.071574 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:40.072886 19993 tablet_copy_client.cc:523] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:40.073756 20223 tablet_replica.cc:330] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf: stopping tablet replica
I20240502 14:00:40.073907 20223 raft_consensus.cc:2229] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:40.074136 20223 raft_consensus.cc:2258] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 14:00:40.074177 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:40.074473 19993 tablet_bootstrap.cc:492] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:40.078876 20227 ts_tablet_manager.cc:902] T bb9f100600a246708457280d91826fce P 4d0f8f311e834d81bd54b9f8e7923bcd: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:00:40.078974 20223 ts_tablet_manager.cc:1875] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:40.079627 20227 tablet_copy_client.cc:308] T bb9f100600a246708457280d91826fce P 4d0f8f311e834d81bd54b9f8e7923bcd: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:00:40.080358 15590 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet bb9f100600a246708457280d91826fce from peer 4d0f8f311e834d81bd54b9f8e7923bcd ({username='slave'} at 127.8.86.132:50891)
I20240502 14:00:40.080441 15590 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet bb9f100600a246708457280d91826fce from peer 4d0f8f311e834d81bd54b9f8e7923bcd at {username='slave'} at 127.8.86.132:50891: session id = 4d0f8f311e834d81bd54b9f8e7923bcd-bb9f100600a246708457280d91826fce
I20240502 14:00:40.080828 19794 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } } }
I20240502 14:00:40.081324 20223 ts_tablet_manager.cc:1888] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240502 14:00:40.081351 15590 tablet_copy_source_session.cc:205] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:40.081389 20223 log.cc:1196] T a7981c3f46d8483caae82fbf2296e762 P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting WAL directory at /tmp/dist-test-task4LMC7s/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714658379161626-8538-0/raft_consensus-itest-cluster/ts-0/wal/wals/a7981c3f46d8483caae82fbf2296e762
I20240502 14:00:40.083386 15415 catalog_manager.cc:4830] TS 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237): tablet a7981c3f46d8483caae82fbf2296e762 (table rebalance_test_table_2 [id=b36f1f16cb3d4b8bae9303e2976dc2e7]) successfully deleted
I20240502 14:00:40.085417 15423 catalog_manager.cc:5484] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 8 to 9, NON_VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:40.085673 15831 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } } }
I20240502 14:00:40.087582 17248 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } } }
I20240502 14:00:40.098717 20227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9f100600a246708457280d91826fce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:40.099372 15831 consensus_queue.cc:237] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
W20240502 14:00:40.101302 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): Couldn't send request to peer 8484181c4ece483c8b408817fab1a917. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:40.101964 17248 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 14:00:40.102373 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:40.102433 18506 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:40.105290 18506 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } } }
I20240502 14:00:40.109114 20227 tablet_copy_client.cc:783] T bb9f100600a246708457280d91826fce P 4d0f8f311e834d81bd54b9f8e7923bcd: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:40.109333 20227 tablet_copy_client.cc:647] T bb9f100600a246708457280d91826fce P 4d0f8f311e834d81bd54b9f8e7923bcd: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:40.111927 15426 catalog_manager.cc:5484] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 9 to 10, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:40.106098 17248 raft_consensus.cc:2932] T 02d3da05d82340178e27218c2ce56b67 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } } }
I20240502 14:00:40.115283 20227 tablet_copy_client.cc:523] T bb9f100600a246708457280d91826fce P 4d0f8f311e834d81bd54b9f8e7923bcd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:40.117339 20227 tablet_bootstrap.cc:492] T bb9f100600a246708457280d91826fce P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
W20240502 14:00:40.134240 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:40.142601 19993 tablet_bootstrap.cc:492] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:40.142972 19993 tablet_bootstrap.cc:492] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: Bootstrap complete.
I20240502 14:00:40.143172 19993 ts_tablet_manager.cc:1367] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.069s user 0.031s sys 0.000s
I20240502 14:00:40.143725 19993 raft_consensus.cc:348] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:00:40.143851 19993 raft_consensus.cc:729] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: LEARNER
W20240502 14:00:40.184429 15899 connection.cc:535] server connection from 127.8.86.134:37349 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:00:40.184856 15414 connection.cc:535] server connection from 127.8.86.134:48229 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:00:40.185117 15504 connection.cc:535] client connection to 127.8.86.134:34379 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:00:40.192611 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:40.192677 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:40.192705 15504 consensus_peers.cc:462] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:40.185508 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:00:40.143959 19993 consensus_queue.cc:260] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:00:40.193197 19993 ts_tablet_manager.cc:1398] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.050s user 0.002s sys 0.000s
W20240502 14:00:40.184584 15635 connection.cc:535] client connection to 127.8.86.134:34379 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:00:40.185009 15897 connection.cc:535] client connection to 127.8.86.134:34379 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:00:40.186419 15964 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6e76bd0ecd3e4c168e141fb7984af3ec"
dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52788
I20240502 14:00:40.193939 15964 raft_consensus.cc:595] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:40.190104 20227 tablet_bootstrap.cc:492] T bb9f100600a246708457280d91826fce P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240502 14:00:40.196051 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:40.196125 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:40.196244 20227 tablet_bootstrap.cc:492] T bb9f100600a246708457280d91826fce P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap complete.
I20240502 14:00:40.196394 15590 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session 8484181c4ece483c8b408817fab1a917-dfdcf1124e8a474facc14c45e8963568 received from {username='slave'} at 127.8.86.130:49841
I20240502 14:00:40.196434 20227 ts_tablet_manager.cc:1367] T bb9f100600a246708457280d91826fce P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.079s user 0.037s sys 0.000s
I20240502 14:00:40.196465 15590 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session 8484181c4ece483c8b408817fab1a917-dfdcf1124e8a474facc14c45e8963568 on tablet dfdcf1124e8a474facc14c45e8963568 with peer 8484181c4ece483c8b408817fab1a917
W20240502 14:00:40.196800 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:40.196907 20227 raft_consensus.cc:348] T bb9f100600a246708457280d91826fce P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } }
I20240502 14:00:40.197060 20227 raft_consensus.cc:729] T bb9f100600a246708457280d91826fce P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: LEARNER
I20240502 14:00:40.197180 20227 consensus_queue.cc:260] T bb9f100600a246708457280d91826fce P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } }
I20240502 14:00:40.197587 20227 ts_tablet_manager.cc:1398] T bb9f100600a246708457280d91826fce P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:40.199151 15590 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session 4d0f8f311e834d81bd54b9f8e7923bcd-bb9f100600a246708457280d91826fce received from {username='slave'} at 127.8.86.132:50891
I20240502 14:00:40.199229 15590 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session 4d0f8f311e834d81bd54b9f8e7923bcd-bb9f100600a246708457280d91826fce on tablet bb9f100600a246708457280d91826fce with peer 4d0f8f311e834d81bd54b9f8e7923bcd
I20240502 14:00:40.204939 15964 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "387bdd8ae3b34f588e2170b061a5bc05"
dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52810
I20240502 14:00:40.205073 15964 raft_consensus.cc:595] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:40.188418 20052 tablet_bootstrap.cc:492] T b0cd21dc897b41aca9118be6df48cc0c P f688370715454d6e814a78fcf85ac5c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:40.215814 20052 tablet_bootstrap.cc:492] T b0cd21dc897b41aca9118be6df48cc0c P f688370715454d6e814a78fcf85ac5c3: Bootstrap complete.
I20240502 14:00:40.216044 20052 ts_tablet_manager.cc:1367] T b0cd21dc897b41aca9118be6df48cc0c P f688370715454d6e814a78fcf85ac5c3: Time spent bootstrapping tablet: real 0.155s user 0.031s sys 0.000s
I20240502 14:00:40.216497 20052 raft_consensus.cc:348] T b0cd21dc897b41aca9118be6df48cc0c P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:40.216609 20052 raft_consensus.cc:729] T b0cd21dc897b41aca9118be6df48cc0c P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f688370715454d6e814a78fcf85ac5c3, State: Initialized, Role: LEARNER
I20240502 14:00:40.216701 20052 consensus_queue.cc:260] T b0cd21dc897b41aca9118be6df48cc0c P f688370715454d6e814a78fcf85ac5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:00:40.217000 20052 ts_tablet_manager.cc:1398] T b0cd21dc897b41aca9118be6df48cc0c P f688370715454d6e814a78fcf85ac5c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:40.219640 17696 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task4LMC7s/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 14:00:34.059865 17702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:34.060138 17702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:34.064882 17702 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:00:35.177991 17702 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 14:00:35.868635 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fc947d3d4f264291a81e7e95cb84f48a: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:35.868732 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:35.868767 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6dcd44a470ab434a988eea52c3916f26: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:35.868842 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b36f1f16cb3d4b8bae9303e2976dc2e7: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:35.868875 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a148b6d170714c99a6024d0a3efa85c8: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:35.868935 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fcc590955678424ab1c27b0b23f75544: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:35.868973 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fc947d3d4f264291a81e7e95cb84f48a: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:35.869012 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:35.869048 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6dcd44a470ab434a988eea52c3916f26: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:35.869079 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af7e16dce074d40ae6277d41e034322: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:35.869117 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 17d53f89ea7b4f918ed8bb35d44f6401: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:35.869156 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b36f1f16cb3d4b8bae9303e2976dc2e7: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:35.869192 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a148b6d170714c99a6024d0a3efa85c8: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:35.869231 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 12389884a03b40f68c532a06ec761e9c: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:35.869271 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 278d2359eb234a2b8e569cd993a004a7: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:35.869320 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fc947d3d4f264291a81e7e95cb84f48a: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:35.869359 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:35.869396 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6dcd44a470ab434a988eea52c3916f26: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:35.869436 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af7e16dce074d40ae6277d41e034322: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:35.869478 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 17d53f89ea7b4f918ed8bb35d44f6401: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:35.926708 17702 rebalancer_tool.cc:1074] tablet 387bdd8ae3b34f588e2170b061a5bc05: '82d7b7201cfc431d90baca1ad0a397cf' -> '3c3658c946994e2ba9f41b64aa327d25' move scheduled
I20240502 14:00:35.986604 17702 rebalancer_tool.cc:1074] tablet 7c57cab68b764db0917ed98e74757ff0: '8484181c4ece483c8b408817fab1a917' -> '8fea283177394bd28d7eacf4fb744d59' move scheduled
I20240502 14:00:36.068337 17702 rebalancer_tool.cc:1074] tablet dfdcf1124e8a474facc14c45e8963568: '4d0f8f311e834d81bd54b9f8e7923bcd' -> 'f688370715454d6e814a78fcf85ac5c3' move scheduled
I20240502 14:00:36.137925 17702 rebalancer_tool.cc:1074] tablet e48dda3e62e14e46b03d79648e3e2773: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '2254b3ee3db647598d781e467584f78d' move scheduled
I20240502 14:00:36.168193 17702 rebalancer_tool.cc:1074] tablet 9870bc797f184a6284ff6ca4f78b3826: '82d7b7201cfc431d90baca1ad0a397cf' -> '3c3658c946994e2ba9f41b64aa327d25' move scheduled
I20240502 14:00:36.261198 17702 rebalancer_tool.cc:1074] tablet 2d92e5885bf94c3e8e0e4dee40017699: '8484181c4ece483c8b408817fab1a917' -> '8fea283177394bd28d7eacf4fb744d59' move scheduled
I20240502 14:00:36.350143 17702 rebalancer_tool.cc:1084] tablet 48fbc670ec504d26a11bf1b9fbd78981: '4d0f8f311e834d81bd54b9f8e7923bcd' -> 'f688370715454d6e814a78fcf85ac5c3' move ignored: Illegal state: tablet 48fbc670ec504d26a11bf1b9fbd78981: replica f688370715454d6e814a78fcf85ac5c3 is already present
I20240502 14:00:36.350242 17702 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:00:37.147312 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af7e16dce074d40ae6277d41e034322: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:37.147389 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:37.147473 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 17d53f89ea7b4f918ed8bb35d44f6401: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:37.147526 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b36f1f16cb3d4b8bae9303e2976dc2e7: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:37.147601 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af7e16dce074d40ae6277d41e034322: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:37.147650 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6dcd44a470ab434a988eea52c3916f26: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:37.147692 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a148b6d170714c99a6024d0a3efa85c8: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:37.147737 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 278d2359eb234a2b8e569cd993a004a7: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:37.147781 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 12389884a03b40f68c532a06ec761e9c: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:37.147815 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fc947d3d4f264291a81e7e95cb84f48a: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:37.147859 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:37.147918 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 17d53f89ea7b4f918ed8bb35d44f6401: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:37.147966 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b36f1f16cb3d4b8bae9303e2976dc2e7: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:37.148005 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af7e16dce074d40ae6277d41e034322: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:37.148059 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 12389884a03b40f68c532a06ec761e9c: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 2254b3ee3db647598d781e467584f78d
I20240502 14:00:37.336016 17702 rebalancer_tool.cc:1074] tablet a7981c3f46d8483caae82fbf2296e762: '4d0f8f311e834d81bd54b9f8e7923bcd' -> '3c3658c946994e2ba9f41b64aa327d25' move scheduled
I20240502 14:00:37.469838 17702 rebalancer_tool.cc:1074] tablet 837e8b1235a44131bdaef24b763d1cac: 'f688370715454d6e814a78fcf85ac5c3' -> '2254b3ee3db647598d781e467584f78d' move scheduled
I20240502 14:00:37.644395 17702 rebalancer_tool.cc:1074] tablet e51db70f5f284c4da3c1669fddb41b00: '1eb0e3a6c782469fa8110f6bb6a585cf' -> 'f688370715454d6e814a78fcf85ac5c3' move scheduled
I20240502 14:00:37.847265 17702 rebalancer_tool.cc:1074] tablet b6d365c10c76416a944ec8fe409e3dc5: '82d7b7201cfc431d90baca1ad0a397cf' -> '2254b3ee3db647598d781e467584f78d' move scheduled
I20240502 14:00:38.212337 17702 rebalancer_tool.cc:1084] tablet d81f14b7c1d1468196d0fe6d67fe93a7: '8484181c4ece483c8b408817fab1a917' -> '2254b3ee3db647598d781e467584f78d' move ignored: Illegal state: tablet d81f14b7c1d1468196d0fe6d67fe93a7: replica 2254b3ee3db647598d781e467584f78d is already present
I20240502 14:00:38.212411 17702 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:00:38.941547 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6dcd44a470ab434a988eea52c3916f26: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 8484181c4ece483c8b408817fab1a917
W20240502 14:00:38.941654 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 8484181c4ece483c8b408817fab1a917
W20240502 14:00:38.941753 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b36f1f16cb3d4b8bae9303e2976dc2e7: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 4d0f8f311e834d81bd54b9f8e7923bcd
W20240502 14:00:38.941835 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a148b6d170714c99a6024d0a3efa85c8: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:38.941897 17702 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6dcd44a470ab434a988eea52c3916f26: could not find any suitable replica to move from server f688370715454d6e814a78fcf85ac5c3 to server 8fea283177394bd28d7eacf4fb744d59
I20240502 14:00:38.979250 17702 rebalancer_tool.cc:1074] tablet 1f80c8590e994a10a9b7c6c174e28173: '1eb0e3a6c782469fa8110f6bb6a585cf' -> 'f688370715454d6e814a78fcf85ac5c3' move scheduled
I20240502 14:00:39.021477 17702 rebalancer_tool.cc:1074] tablet 2886dfbec1e141198b0cf7a435fac498: '4d0f8f311e834d81bd54b9f8e7923bcd' -> '8fea283177394bd28d7eacf4fb744d59' move scheduled
I20240502 14:00:39.128607 17702 rebalancer_tool.cc:1074] tablet 9391f16ae2d247ec8a71e5c971cdda71: '8484181c4ece483c8b408817fab1a917' -> '2254b3ee3db647598d781e467584f78d' move scheduled
I20240502 14:00:39.235071 17702 rebalancer_tool.cc:1074] tablet bb9f100600a246708457280d91826fce: '82d7b7201cfc431d90baca1ad0a397cf' -> '3c3658c946994e2ba9f41b64aa327d25' move scheduled
I20240502 14:00:39.250649 17702 rebalancer_tool.cc:1084] tablet b0cd21dc897b41aca9118be6df48cc0c: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '2254b3ee3db647598d781e467584f78d' move ignored: Illegal state: tablet b0cd21dc897b41aca9118be6df48cc0c: replica 2254b3ee3db647598d781e467584f78d is already present
I20240502 14:00:39.250735 17702 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:00:39.985325 20141 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:00:39.811628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.86.136:38631 (local address 127.0.0.1:37666)
0502 14:00:39.813923 (+ 2295us) negotiation.cc:107] Waiting for socket to connect
0502 14:00:39.813937 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0502 14:00:39.814017 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:00:39.984583 (+170566us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2230,"thread_start_us":116,"threads_started":1}
W20240502 14:00:40.185320 20141 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:00:40.184432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.86.134:34379 (local address 127.0.0.1:37246)
0502 14:00:40.185173 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0502 14:00:40.185225 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":631}
Illegal state: tablet server 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): unacceptable health status UNAVAILABLE
I20240502 14:00:40.221652 15590 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session f688370715454d6e814a78fcf85ac5c3-b0cd21dc897b41aca9118be6df48cc0c received from {username='slave'} at 127.8.86.133:50351
I20240502 14:00:40.221753 15590 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session f688370715454d6e814a78fcf85ac5c3-b0cd21dc897b41aca9118be6df48cc0c on tablet b0cd21dc897b41aca9118be6df48cc0c with peer f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:40.230970 15504 consensus_peers.cc:462] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
I20240502 14:00:40.241945 19285 raft_consensus.cc:979] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd: : Instructing follower f688370715454d6e814a78fcf85ac5c3 to start an election
I20240502 14:00:40.242046 19285 raft_consensus.cc:1067] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Signalling peer f688370715454d6e814a78fcf85ac5c3 to start an election
I20240502 14:00:40.242713 17248 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6e76bd0ecd3e4c168e141fb7984af3ec"
dest_uuid: "f688370715454d6e814a78fcf85ac5c3"
from {username='slave'} at 127.8.86.132:37141
I20240502 14:00:40.242882 17248 raft_consensus.cc:482] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:40.243003 17248 raft_consensus.cc:3037] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:40.244213 17248 raft_consensus.cc:504] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:40.246935 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e76bd0ecd3e4c168e141fb7984af3ec" candidate_uuid: "f688370715454d6e814a78fcf85ac5c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
I20240502 14:00:40.247074 15964 raft_consensus.cc:3032] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:40.247124 15964 raft_consensus.cc:729] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Running, Role: LEADER
I20240502 14:00:40.247294 15964 consensus_queue.cc:260] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:40.247483 15964 raft_consensus.cc:3037] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:40.248598 15964 raft_consensus.cc:2454] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f688370715454d6e814a78fcf85ac5c3 in term 2.
I20240502 14:00:40.249162 15702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e76bd0ecd3e4c168e141fb7984af3ec" candidate_uuid: "f688370715454d6e814a78fcf85ac5c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "8484181c4ece483c8b408817fab1a917"
I20240502 14:00:40.249287 15702 raft_consensus.cc:3037] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:00:40.250528 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:00:40.251308 15702 raft_consensus.cc:2454] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f688370715454d6e814a78fcf85ac5c3 in term 2.
I20240502 14:00:40.251745 17184 leader_election.cc:302] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 8484181c4ece483c8b408817fab1a917, f688370715454d6e814a78fcf85ac5c3; no voters:
I20240502 14:00:40.252022 19378 raft_consensus.cc:2781] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:40.252210 19378 raft_consensus.cc:686] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 [term 2 LEADER]: Becoming Leader. State: Replica: f688370715454d6e814a78fcf85ac5c3, State: Running, Role: LEADER
I20240502 14:00:40.252498 19378 consensus_queue.cc:237] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:40.250344 17248 leader_election.cc:288] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 [CANDIDATE]: Term 2 election: Requested vote from peers 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719), 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261), 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245)
I20240502 14:00:40.257403 15423 catalog_manager.cc:5484] T 6e76bd0ecd3e4c168e141fb7984af3ec P f688370715454d6e814a78fcf85ac5c3 reported cstate change: term changed from 1 to 2, leader changed from 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) to f688370715454d6e814a78fcf85ac5c3 (127.8.86.133). New cstate: current_term: 2 leader_uuid: "f688370715454d6e814a78fcf85ac5c3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:40.273340 17514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e76bd0ecd3e4c168e141fb7984af3ec" candidate_uuid: "f688370715454d6e814a78fcf85ac5c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "8fea283177394bd28d7eacf4fb744d59"
I20240502 14:00:40.273555 17514 raft_consensus.cc:3037] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:40.274889 17514 raft_consensus.cc:2454] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f688370715454d6e814a78fcf85ac5c3 in term 2.
I20240502 14:00:40.276818 18402 raft_consensus.cc:1050] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf: attempting to promote NON_VOTER 8fea283177394bd28d7eacf4fb744d59 to VOTER
I20240502 14:00:40.277330 18402 consensus_queue.cc:237] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:40.281143 15702 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:00:40.283694 17514 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:00:40.284202 19920 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:00:40.285017 19920 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 14:00:40.285681 15504 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:40.287755 15504 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:40.287856 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:40.289042 17231 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:40.291658 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:40.296957 15681 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:40.299460 19920 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:40.300981 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:40.301925 17515 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:40.302186 15702 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:40.307237 15425 catalog_manager.cc:5484] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 10 to 11, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:00:40.307976 15549 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:00:40.319869 15897 consensus_peers.cc:485] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:40.319917 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:40.320402 17248 raft_consensus.cc:1203] T b0cd21dc897b41aca9118be6df48cc0c P f688370715454d6e814a78fcf85ac5c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240502 14:00:40.320947 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:40.330317 15944 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:40.361974 20052 ts_tablet_manager.cc:902] T 1a5e33f337bd47b2a6b11b29a5eacad8 P f688370715454d6e814a78fcf85ac5c3: Initiating tablet copy from peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
F20240502 14:00:40.362185 20052 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-02d3da05d82340178e27218c2ce56b67.queue_time_us, tablet-download-pool-02d3da05d82340178e27218c2ce56b67.queue_time_us) (tablet-download-pool-02d3da05d82340178e27218c2ce56b67.run_cpu_time_us, tablet-download-pool-02d3da05d82340178e27218c2ce56b67.run_cpu_time_us) (tablet-download-pool-02d3da05d82340178e27218c2ce56b67.run_wall_time_us, tablet-download-pool-02d3da05d82340178e27218c2ce56b67.run_wall_time_us) (tablet-download-pool-1a5e33f337bd47b2a6b11b29a5eacad8.queue_time_us, tablet-download-pool-1a5e33f337bd47b2a6b11b29a5eacad8.queue_time_us) (tablet-download-pool-1c004727e7c64f33bc9492184b77fde9.queue_time_us, tablet-download-pool-1c004727e7c64f33bc9492184b77fde9.queue_time_us) (tablet-download-pool-1c004727e7c64f33bc9492184b77fde9.run_cpu_time_us, tablet-download-pool-1c004727e7c64f33bc9492184b77fde9.run_cpu_time_us) (tablet-download-pool-1c004727e7c64f33bc9492184b77fde9.run_wall_time_us, tablet-download-pool-1c004727e7c64f33bc9492184b77fde9.run_wall_time_us) (tablet-download-pool-1f562ad00ec34a8d9184671ed55d84d1.queue_time_us, tablet-download-pool-1f562ad00ec34a8d9184671ed55d84d1.queue_time_us) (tablet-download-pool-1f562ad00ec34a8d9184671ed55d84d1.run_cpu_time_us, tablet-download-pool-1f562ad00ec34a8d9184671ed55d84d1.run_cpu_time_us) (tablet-download-pool-1f562ad00ec34a8d9184671ed55d84d1.run_wall_time_us, tablet-download-pool-1f562ad00ec34a8d9184671ed55d84d1.run_wall_time_us) (tablet-download-pool-1f80c8590e994a10a9b7c6c174e28173.queue_time_us, tablet-download-pool-1f80c8590e994a10a9b7c6c174e28173.queue_time_us) (tablet-download-pool-1f80c8590e994a10a9b7c6c174e28173.run_cpu_time_us, tablet-download-pool-1f80c8590e994a10a9b7c6c174e28173.run_cpu_time_us) (tablet-download-pool-1f80c8590e994a10a9b7c6c174e28173.run_wall_time_us, tablet-download-pool-1f80c8590e994a10a9b7c6c174e28173.run_wall_time_us) (tablet-download-pool-22eef55211ca4f65979022d2f4ae829f.queue_time_us, tablet-download-pool-22eef55211ca4f65979022d2f4ae829f.queue_time_us) (tablet-download-pool-22eef55211ca4f65979022d2f4ae829f.run_cpu_time_us, tablet-download-pool-22eef55211ca4f65979022d2f4ae829f.run_cpu_time_us) (tablet-download-pool-22eef55211ca4f65979022d2f4ae829f.run_wall_time_us, tablet-download-pool-22eef55211ca4f65979022d2f4ae829f.run_wall_time_us) (tablet-download-pool-2886dfbec1e141198b0cf7a435fac498.queue_time_us, tablet-download-pool-2886dfbec1e141198b0cf7a435fac498.queue_time_us) (tablet-download-pool-2886dfbec1e141198b0cf7a435fac498.run_cpu_time_us, tablet-download-pool-2886dfbec1e141198b0cf7a435fac498.run_cpu_time_us) (tablet-download-pool-2886dfbec1e141198b0cf7a435fac498.run_wall_time_us, tablet-download-pool-2886dfbec1e141198b0cf7a435fac498.run_wall_time_us) (tablet-download-pool-387bdd8ae3b34f588e2170b061a5bc05.queue_time_us, tablet-download-pool-387bdd8ae3b34f588e2170b061a5bc05.queue_time_us) (tablet-download-pool-387bdd8ae3b34f588e2170b061a5bc05.run_cpu_time_us, tablet-download-pool-387bdd8ae3b34f588e2170b061a5bc05.run_cpu_time_us) (tablet-download-pool-387bdd8ae3b34f588e2170b061a5bc05.run_wall_time_us, tablet-download-pool-387bdd8ae3b34f588e2170b061a5bc05.run_wall_time_us) (tablet-download-pool-48fbc670ec504d26a11bf1b9fbd78981.queue_time_us, tablet-download-pool-48fbc670ec504d26a11bf1b9fbd78981.queue_time_us) (tablet-download-pool-48fbc670ec504d26a11bf1b9fbd78981.run_cpu_time_us, tablet-download-pool-48fbc670ec504d26a11bf1b9fbd78981.run_cpu_time_us) (tablet-download-pool-48fbc670ec504d26a11bf1b9fbd78981.run_wall_time_us, tablet-download-pool-48fbc670ec504d26a11bf1b9fbd78981.run_wall_time_us) (tablet-download-pool-6e76bd0ecd3e4c168e141fb7984af3ec.queue_time_us, tablet-download-pool-6e76bd0ecd3e4c168e141fb7984af3ec.queue_time_us) (tablet-download-pool-6e76bd0ecd3e4c168e141fb7984af3ec.run_cpu_time_us, tablet-download-pool-6e76bd0ecd3e4c168e141fb7984af3ec.run_cpu_time_us) (tablet-download-pool-6e76bd0ecd3e4c168e141fb7984af3ec.run_wall_time_us, tablet-download-pool-6e76bd0ecd3e4c168e141fb7984af3ec.run_wall_time_us) (tablet-download-pool-837e8b1235a44131bdaef24b763d1cac.queue_time_us, tablet-download-pool-837e8b1235a44131bdaef24b763d1cac.queue_time_us) (tablet-download-pool-837e8b1235a44131bdaef24b763d1cac.run_cpu_time_us, tablet-download-pool-837e8b1235a44131bdaef24b763d1cac.run_cpu_time_us) (tablet-download-pool-837e8b1235a44131bdaef24b763d1cac.run_wall_time_us, tablet-download-pool-837e8b1235a44131bdaef24b763d1cac.run_wall_time_us) (tablet-download-pool-b07f4db6f82e4a56accdbf1bbbc5f965.queue_time_us, tablet-download-pool-b07f4db6f82e4a56accdbf1bbbc5f965.queue_time_us) (tablet-download-pool-b07f4db6f82e4a56accdbf1bbbc5f965.run_cpu_time_us, tablet-download-pool-b07f4db6f82e4a56accdbf1bbbc5f965.run_cpu_time_us) (tablet-download-pool-b07f4db6f82e4a56accdbf1bbbc5f965.run_wall_time_us, tablet-download-pool-b07f4db6f82e4a56accdbf1bbbc5f965.run_wall_time_us) (tablet-download-pool-b0cd21dc897b41aca9118be6df48cc0c.queue_time_us, tablet-download-pool-b0cd21dc897b41aca9118be6df48cc0c.queue_time_us) (tablet-download-pool-b0cd21dc897b41aca9118be6df48cc0c.run_cpu_time_us, tablet-download-pool-b0cd21dc897b41aca9118be6df48cc0c.run_cpu_time_us) (tablet-download-pool-b0cd21dc897b41aca9118be6df48cc0c.run_wall_time_us, tablet-download-pool-b0cd21dc897b41aca9118be6df48cc0c.run_wall_time_us) (tablet-download-pool-dfdcf1124e8a474facc14c45e8963568.queue_time_us, tablet-download-pool-dfdcf1124e8a474facc14c45e8963568.queue_time_us) (tablet-download-pool-dfdcf1124e8a474facc14c45e8963568.run_cpu_time_us, tablet-download-pool-dfdcf1124e8a474facc14c45e8963568.run_cpu_time_us) (tablet-download-pool-dfdcf1124e8a474facc14c45e8963568.run_wall_time_us, tablet-download-pool-dfdcf1124e8a474facc14c45e8963568.run_wall_time_us) (tablet-download-pool-e51db70f5f284c4da3c1669fddb41b00.queue_time_us, tablet-download-pool-e51db70f5f284c4da3c1669fddb41b00.queue_time_us) (tablet-download-pool-e51db70f5f284c4da3c1669fddb41b00.run_cpu_time_us, tablet-download-pool-e51db70f5f284c4da3c1669fddb41b00.run_cpu_time_us) (tablet-download-pool-e51db70f5f284c4da3c1669fddb41b00.run_wall_time_us, tablet-download-pool-e51db70f5f284c4da3c1669fddb41b00.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f32675c9dcd google::LogMessage::Fail() at ??:0
I20240502 14:00:40.375312 18566 raft_consensus.cc:979] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd: : Instructing follower f688370715454d6e814a78fcf85ac5c3 to start an election
I20240502 14:00:40.375417 18566 raft_consensus.cc:1067] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Signalling peer f688370715454d6e814a78fcf85ac5c3 to start an election
I20240502 14:00:40.375859 17699 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task4LMC7s/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 14:00:34.163712 17705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:34.164036 17705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:34.168408 17705 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:00:35.266463 17705 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 14:00:35.723932 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:35.724169 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 17d53f89ea7b4f918ed8bb35d44f6401: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:35.724239 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af7e16dce074d40ae6277d41e034322: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:35.724296 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:35.724382 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fc947d3d4f264291a81e7e95cb84f48a: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:35.724471 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 278d2359eb234a2b8e569cd993a004a7: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:35.724529 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6dcd44a470ab434a988eea52c3916f26: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:35.725004 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b36f1f16cb3d4b8bae9303e2976dc2e7: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:35.725116 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 17d53f89ea7b4f918ed8bb35d44f6401: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:35.725237 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a148b6d170714c99a6024d0a3efa85c8: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:35.725319 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fcc590955678424ab1c27b0b23f75544: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:35.725386 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 12389884a03b40f68c532a06ec761e9c: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:35.725467 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 278d2359eb234a2b8e569cd993a004a7: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:35.725625 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af7e16dce074d40ae6277d41e034322: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:35.725693 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6dcd44a470ab434a988eea52c3916f26: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:35.725737 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b36f1f16cb3d4b8bae9303e2976dc2e7: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:35.803165 17705 rebalancer_tool.cc:1074] tablet 22eef55211ca4f65979022d2f4ae829f: '8484181c4ece483c8b408817fab1a917' -> '2254b3ee3db647598d781e467584f78d' move scheduled
I20240502 14:00:35.856474 17705 rebalancer_tool.cc:1074] tablet dfdcf1124e8a474facc14c45e8963568: '4d0f8f311e834d81bd54b9f8e7923bcd' -> '3c3658c946994e2ba9f41b64aa327d25' move scheduled
I20240502 14:00:35.898419 17705 rebalancer_tool.cc:1074] tablet d81f14b7c1d1468196d0fe6d67fe93a7: '82d7b7201cfc431d90baca1ad0a397cf' -> '8fea283177394bd28d7eacf4fb744d59' move scheduled
I20240502 14:00:35.982921 17705 rebalancer_tool.cc:1074] tablet 1c004727e7c64f33bc9492184b77fde9: '1eb0e3a6c782469fa8110f6bb6a585cf' -> 'f688370715454d6e814a78fcf85ac5c3' move scheduled
I20240502 14:00:36.058266 17705 rebalancer_tool.cc:1084] tablet 1f562ad00ec34a8d9184671ed55d84d1: '8484181c4ece483c8b408817fab1a917' -> '2254b3ee3db647598d781e467584f78d' move ignored: Illegal state: tablet 1f562ad00ec34a8d9184671ed55d84d1: replica 2254b3ee3db647598d781e467584f78d is already present
I20240502 14:00:36.058383 17705 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:00:36.705430 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:36.705730 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fc947d3d4f264291a81e7e95cb84f48a: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:36.705786 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af7e16dce074d40ae6277d41e034322: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:36.705881 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af7e16dce074d40ae6277d41e034322: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:36.705930 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:36.706053 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fc947d3d4f264291a81e7e95cb84f48a: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:36.726739 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 278d2359eb234a2b8e569cd993a004a7: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:36.726858 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 12389884a03b40f68c532a06ec761e9c: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:36.726959 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 17d53f89ea7b4f918ed8bb35d44f6401: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:36.727010 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af7e16dce074d40ae6277d41e034322: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:36.727073 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b36f1f16cb3d4b8bae9303e2976dc2e7: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:36.727120 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:36.727167 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a148b6d170714c99a6024d0a3efa85c8: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:36.727218 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fcc590955678424ab1c27b0b23f75544: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:36.727263 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fc947d3d4f264291a81e7e95cb84f48a: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:36.727309 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 278d2359eb234a2b8e569cd993a004a7: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:36.727396 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6dcd44a470ab434a988eea52c3916f26: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:36.850950 17705 rebalancer_tool.cc:1074] tablet 48fbc670ec504d26a11bf1b9fbd78981: '4d0f8f311e834d81bd54b9f8e7923bcd' -> '3c3658c946994e2ba9f41b64aa327d25' move scheduled
I20240502 14:00:37.183518 17705 rebalancer_tool.cc:1074] tablet 7c57cab68b764db0917ed98e74757ff0: '8484181c4ece483c8b408817fab1a917' -> '2254b3ee3db647598d781e467584f78d' move scheduled
I20240502 14:00:37.257624 17705 rebalancer_tool.cc:1074] tablet e48dda3e62e14e46b03d79648e3e2773: '82d7b7201cfc431d90baca1ad0a397cf' -> '8fea283177394bd28d7eacf4fb744d59' move scheduled
I20240502 14:00:37.467077 17705 rebalancer_tool.cc:1074] tablet 6e76bd0ecd3e4c168e141fb7984af3ec: '1eb0e3a6c782469fa8110f6bb6a585cf' -> 'f688370715454d6e814a78fcf85ac5c3' move scheduled
I20240502 14:00:37.843374 17705 rebalancer_tool.cc:1084] tablet 28e371c1a999431499e09479e8f5b44b: '4d0f8f311e834d81bd54b9f8e7923bcd' -> '8fea283177394bd28d7eacf4fb744d59' move ignored: Illegal state: tablet 28e371c1a999431499e09479e8f5b44b: replica 8fea283177394bd28d7eacf4fb744d59 is already present
I20240502 14:00:37.843444 17705 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:00:38.552556 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6dcd44a470ab434a988eea52c3916f26: could not find any suitable replica to move from server f688370715454d6e814a78fcf85ac5c3 to server 8484181c4ece483c8b408817fab1a917
W20240502 14:00:38.552748 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a148b6d170714c99a6024d0a3efa85c8: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:38.552783 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 17d53f89ea7b4f918ed8bb35d44f6401: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:38.552811 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 12389884a03b40f68c532a06ec761e9c: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 8fea283177394bd28d7eacf4fb744d59
I20240502 14:00:38.795533 17705 rebalancer_tool.cc:1074] tablet 387bdd8ae3b34f588e2170b061a5bc05: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '' move scheduled
I20240502 14:00:38.870764 17705 rebalancer_tool.cc:1074] tablet 436c127af87c49e8b7bf9b6571f7dc30: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '2254b3ee3db647598d781e467584f78d' move scheduled
I20240502 14:00:38.927784 17705 rebalancer_tool.cc:1074] tablet 2d92e5885bf94c3e8e0e4dee40017699: '4d0f8f311e834d81bd54b9f8e7923bcd' -> '3c3658c946994e2ba9f41b64aa327d25' move scheduled
I20240502 14:00:38.952219 17705 rebalancer_tool.cc:1084] tablet 9391f16ae2d247ec8a71e5c971cdda71: '8484181c4ece483c8b408817fab1a917' -> '3c3658c946994e2ba9f41b64aa327d25' move ignored: Illegal state: tablet 9391f16ae2d247ec8a71e5c971cdda71: replica 3c3658c946994e2ba9f41b64aa327d25 is already present
I20240502 14:00:38.952301 17705 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:00:39.996201 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6dcd44a470ab434a988eea52c3916f26: could not find any suitable replica to move from server f688370715454d6e814a78fcf85ac5c3 to server 8484181c4ece483c8b408817fab1a917
W20240502 14:00:39.996304 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b36f1f16cb3d4b8bae9303e2976dc2e7: could not find any suitable replica to move from server 3c3658c946994e2ba9f41b64aa327d25 to server 4d0f8f311e834d81bd54b9f8e7923bcd
W20240502 14:00:39.996429 17705 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fcc590955678424ab1c27b0b23f75544: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 8484181c4ece483c8b408817fab1a917
I20240502 14:00:40.112738 17705 rebalancer_tool.cc:1074] tablet 02d3da05d82340178e27218c2ce56b67: '2254b3ee3db647598d781e467584f78d' -> '8484181c4ece483c8b408817fab1a917' move scheduled
I20240502 14:00:40.124132 17705 rebalancer_tool.cc:1084] tablet 1a5e33f337bd47b2a6b11b29a5eacad8: '82d7b7201cfc431d90baca1ad0a397cf' -> 'f688370715454d6e814a78fcf85ac5c3' move ignored: Illegal state: tablet 1a5e33f337bd47b2a6b11b29a5eacad8: replica f688370715454d6e814a78fcf85ac5c3 is already present
I20240502 14:00:40.124212 17705 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): unacceptable health status UNAVAILABLE
I20240502 14:00:40.376075 17248 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "387bdd8ae3b34f588e2170b061a5bc05"
dest_uuid: "f688370715454d6e814a78fcf85ac5c3"
from {username='slave'} at 127.8.86.132:37141
I20240502 14:00:40.376202 17248 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:40.376266 17248 raft_consensus.cc:3037] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:40.377383 17248 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:40.377877 17248 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [CANDIDATE]: Term 2 election: Requested vote from peers 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719), 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379)
W20240502 14:00:40.378445 17184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:00:40.378590 15964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387bdd8ae3b34f588e2170b061a5bc05" candidate_uuid: "f688370715454d6e814a78fcf85ac5c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
I20240502 14:00:40.378728 15964 raft_consensus.cc:3032] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:40.378783 15964 raft_consensus.cc:729] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Running, Role: LEADER
@ 0x7f32675cdb93 google::LogMessage::SendToLog() at ??:0
I20240502 14:00:40.378960 15964 consensus_queue.cc:260] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:40.379230 15964 raft_consensus.cc:3037] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:00:40.379355 17184 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:00:40.380475 15964 raft_consensus.cc:2454] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f688370715454d6e814a78fcf85ac5c3 in term 2.
I20240502 14:00:40.380833 17184 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, f688370715454d6e814a78fcf85ac5c3; no voters: 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:40.381036 19378 raft_consensus.cc:2781] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:40.381100 19378 raft_consensus.cc:686] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [term 2 LEADER]: Becoming Leader. State: Replica: f688370715454d6e814a78fcf85ac5c3, State: Running, Role: LEADER
I20240502 14:00:40.381227 19378 consensus_queue.cc:237] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
@ 0x7f32675c97cc google::LogMessage::Flush() at ??:0
I20240502 14:00:40.383375 15429 catalog_manager.cc:5484] T 387bdd8ae3b34f588e2170b061a5bc05 P f688370715454d6e814a78fcf85ac5c3 reported cstate change: term changed from 1 to 2, leader changed from 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) to f688370715454d6e814a78fcf85ac5c3 (127.8.86.133). New cstate: current_term: 2 leader_uuid: "f688370715454d6e814a78fcf85ac5c3" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
@ 0x7f32675caf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3268a33044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f3268a17f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f3268a15efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f326b5c877a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f326b5d1a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 14:00:40.403750 15702 raft_consensus.cc:1203] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
@ 0x7f326b65276f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
W20240502 14:00:40.407552 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
@ 0x7f326b6664f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f326b651226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f326b660c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5631d540375c std::function<>::operator()() at ??:0
@ 0x7f3268a1ae06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3268a1b6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3268a1d03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5631d540375c std::function<>::operator()() at ??:0
@ 0x7f3268a0a807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f32692486db start_thread at ??:0
@ 0x7f326678071f clone at ??:0
I20240502 14:00:40.460007 15569 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "436c127af87c49e8b7bf9b6571f7dc30"
dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41424
I20240502 14:00:40.460206 15569 raft_consensus.cc:595] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Received request to transfer leadership
W20240502 14:00:40.464306 15504 consensus_peers.cc:462] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:00:40.465905 15964 raft_consensus.cc:1203] T bb9f100600a246708457280d91826fce P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 14:00:40.466514 20227 ts_tablet_manager.cc:902] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: Initiating tablet copy from peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:00:40.467011 20227 tablet_copy_client.cc:308] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:42261
I20240502 14:00:40.467555 15721 tablet_copy_service.cc:140] P 8484181c4ece483c8b408817fab1a917: Received BeginTabletCopySession request for tablet 48fbc670ec504d26a11bf1b9fbd78981 from peer 4d0f8f311e834d81bd54b9f8e7923bcd ({username='slave'} at 127.8.86.132:50229)
I20240502 14:00:40.467639 15721 tablet_copy_service.cc:161] P 8484181c4ece483c8b408817fab1a917: Beginning new tablet copy session on tablet 48fbc670ec504d26a11bf1b9fbd78981 from peer 4d0f8f311e834d81bd54b9f8e7923bcd at {username='slave'} at 127.8.86.132:50229: session id = 4d0f8f311e834d81bd54b9f8e7923bcd-48fbc670ec504d26a11bf1b9fbd78981
I20240502 14:00:40.468709 15721 tablet_copy_source_session.cc:205] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:40.469481 20227 ts_tablet_manager.cc:1875] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:00:40.471716 20227 ts_tablet_manager.cc:1888] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240502 14:00:40.471901 20227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48fbc670ec504d26a11bf1b9fbd78981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:40.473389 20227 tablet_copy_client.cc:783] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:40.473572 20227 tablet_copy_client.cc:647] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:40.474781 20227 tablet_copy_client.cc:523] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:40.475864 20227 tablet_bootstrap.cc:492] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap starting.
I20240502 14:00:40.494402 19993 ts_tablet_manager.cc:902] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:00:40.494918 19993 tablet_copy_client.cc:308] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
I20240502 14:00:40.495436 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 02d3da05d82340178e27218c2ce56b67 from peer 8484181c4ece483c8b408817fab1a917 ({username='slave'} at 127.8.86.130:49307)
I20240502 14:00:40.495548 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 02d3da05d82340178e27218c2ce56b67 from peer 8484181c4ece483c8b408817fab1a917 at {username='slave'} at 127.8.86.130:49307: session id = 8484181c4ece483c8b408817fab1a917-02d3da05d82340178e27218c2ce56b67
I20240502 14:00:40.496440 15853 tablet_copy_source_session.cc:205] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:40.497229 19993 ts_tablet_manager.cc:1875] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:00:40.501189 19993 ts_tablet_manager.cc:1888] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240502 14:00:40.501435 19993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d3da05d82340178e27218c2ce56b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:40.503021 19993 tablet_copy_client.cc:783] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:40.503381 19993 tablet_copy_client.cc:647] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:40.504652 19993 tablet_copy_client.cc:523] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:40.506265 19993 tablet_bootstrap.cc:492] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:00:40.513613 20250 raft_consensus.cc:1050] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf: attempting to promote NON_VOTER 8fea283177394bd28d7eacf4fb744d59 to VOTER
I20240502 14:00:40.515107 20250 consensus_queue.cc:237] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:00:40.517799 17515 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P 8fea283177394bd28d7eacf4fb744d59 [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:40.517800 15964 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:40.518214 20169 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:40.518800 18506 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:40.523108 20227 tablet_bootstrap.cc:492] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:40.523402 20227 tablet_bootstrap.cc:492] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: Bootstrap complete.
I20240502 14:00:40.523597 20227 ts_tablet_manager.cc:1367] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent bootstrapping tablet: real 0.048s user 0.036s sys 0.000s
I20240502 14:00:40.524099 20227 raft_consensus.cc:348] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } }
I20240502 14:00:40.524240 20227 raft_consensus.cc:729] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4d0f8f311e834d81bd54b9f8e7923bcd, State: Initialized, Role: LEARNER
I20240502 14:00:40.524353 20227 consensus_queue.cc:260] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: NON_VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: true } }
I20240502 14:00:40.525408 20227 ts_tablet_manager.cc:1398] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:40.525957 15721 tablet_copy_service.cc:342] P 8484181c4ece483c8b408817fab1a917: Request end of tablet copy session 4d0f8f311e834d81bd54b9f8e7923bcd-48fbc670ec504d26a11bf1b9fbd78981 received from {username='slave'} at 127.8.86.132:50229
I20240502 14:00:40.526058 15721 tablet_copy_service.cc:433] P 8484181c4ece483c8b408817fab1a917: ending tablet copy session 4d0f8f311e834d81bd54b9f8e7923bcd-48fbc670ec504d26a11bf1b9fbd78981 on tablet 48fbc670ec504d26a11bf1b9fbd78981 with peer 4d0f8f311e834d81bd54b9f8e7923bcd
I20240502 14:00:40.531435 20250 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:40.531877 15964 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:40.534878 15429 catalog_manager.cc:5484] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 9 to 10, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:40.534914 17515 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:00:40.548398 19993 tablet_bootstrap.cc:492] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:40.548750 19993 tablet_bootstrap.cc:492] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: Bootstrap complete.
I20240502 14:00:40.548960 19993 ts_tablet_manager.cc:1367] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.043s user 0.025s sys 0.005s
I20240502 14:00:40.549474 19993 raft_consensus.cc:348] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:00:40.549634 19993 raft_consensus.cc:729] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: LEARNER
I20240502 14:00:40.549757 19993 consensus_queue.cc:260] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:00:40.550063 19993 ts_tablet_manager.cc:1398] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:40.550714 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session 8484181c4ece483c8b408817fab1a917-02d3da05d82340178e27218c2ce56b67 received from {username='slave'} at 127.8.86.130:49307
I20240502 14:00:40.550957 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session 8484181c4ece483c8b408817fab1a917-02d3da05d82340178e27218c2ce56b67 on tablet 02d3da05d82340178e27218c2ce56b67 with peer 8484181c4ece483c8b408817fab1a917
I20240502 14:00:40.588261 18506 raft_consensus.cc:1050] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf: attempting to promote NON_VOTER 8484181c4ece483c8b408817fab1a917 to VOTER
I20240502 14:00:40.588721 18506 consensus_queue.cc:237] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } }
I20240502 14:00:40.590262 17515 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 14:00:40.590427 15702 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 14:00:40.590940 20250 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 14:00:40.591535 20169 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 14:00:40.593209 20250 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:00:40.593655 15702 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:00:40.595224 17515 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:00:40.595825 15426 catalog_manager.cc:5484] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 13 to 14, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:40.599423 15832 consensus_queue.cc:237] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } }
I20240502 14:00:40.600761 17515 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 14:00:40.600847 15702 raft_consensus.cc:1261] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 14:00:40.601183 20169 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 14:00:40.601650 20169 consensus_queue.cc:1035] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 14:00:40.603015 18506 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:00:40.603405 17515 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:00:40.604841 15416 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 22eef55211ca4f65979022d2f4ae829f with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:40.611007 15702 raft_consensus.cc:2932] T 22eef55211ca4f65979022d2f4ae829f P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:00:40.611570 15425 catalog_manager.cc:5484] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 14 to 15, VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:00:40.670226 15414 connection.cc:535] client connection to 127.8.86.133:42151 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:00:40.670444 15414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:00:40.670598 15767 connection.cc:535] client connection to 127.8.86.133:42151 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:00:40.670601 15636 connection.cc:535] client connection to 127.8.86.133:42151 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:00:40.670912 15505 connection.cc:535] client connection to 127.8.86.133:42151 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:00:40.672240 20389 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:00:40.671393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.86.133:42151 (local address 127.8.86.129:42987)
0502 14:00:40.671526 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0502 14:00:40.671541 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0502 14:00:40.671638 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:00:40.671735 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":55}
W20240502 14:00:40.672241 20224 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:00:40.671310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.86.133:42151 (local address 127.8.86.130:57503)
0502 14:00:40.671550 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0502 14:00:40.671566 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0502 14:00:40.671664 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:00:40.671760 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":142}
W20240502 14:00:40.672559 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: BlockingWrite error: write error: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:40.673120 20191 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:00:40.670975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.86.133:42151 (local address 127.8.86.132:34629)
0502 14:00:40.671155 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0502 14:00:40.671168 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0502 14:00:40.671260 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:00:40.672777 (+ 1517us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":98}
W20240502 14:00:40.673422 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:00:40.673481 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:40.673569 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:40.673745 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: BlockingWrite error: write error: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:40.673827 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: BlockingWrite error: write error: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:40.673884 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: BlockingWrite error: write error: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:40.676060 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22eef55211ca4f65979022d2f4ae829f: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:00:40.678896 15569 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "436c127af87c49e8b7bf9b6571f7dc30"
dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41466
I20240502 14:00:40.679044 15569 raft_consensus.cc:595] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Received request to transfer leadership
W20240502 14:00:40.709203 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:40.721282 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:40.753037 19694 raft_consensus.cc:979] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf: : Instructing follower 8484181c4ece483c8b408817fab1a917 to start an election
I20240502 14:00:40.753194 20406 raft_consensus.cc:1067] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Signalling peer 8484181c4ece483c8b408817fab1a917 to start an election
I20240502 14:00:40.754138 15702 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "436c127af87c49e8b7bf9b6571f7dc30"
dest_uuid: "8484181c4ece483c8b408817fab1a917"
from {username='slave'} at 127.8.86.129:32983
I20240502 14:00:40.754324 15702 raft_consensus.cc:482] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:40.754392 15702 raft_consensus.cc:3037] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:40.755554 15702 raft_consensus.cc:504] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:40.756071 15702 leader_election.cc:288] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 2 election: Requested vote from peers 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237), 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245)
I20240502 14:00:40.756429 17515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436c127af87c49e8b7bf9b6571f7dc30" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "8fea283177394bd28d7eacf4fb744d59"
I20240502 14:00:40.756580 17515 raft_consensus.cc:3037] T 436c127af87c49e8b7bf9b6571f7dc30 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:00:40.757092 15635 leader_election.cc:334] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:00:40.757091 15569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436c127af87c49e8b7bf9b6571f7dc30" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf"
I20240502 14:00:40.757205 15569 raft_consensus.cc:3032] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:40.757247 15569 raft_consensus.cc:729] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Running, Role: LEADER
I20240502 14:00:40.757411 15569 consensus_queue.cc:260] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:40.757623 15569 raft_consensus.cc:3037] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:40.757630 17515 raft_consensus.cc:2454] T 436c127af87c49e8b7bf9b6571f7dc30 P 8fea283177394bd28d7eacf4fb744d59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 2.
I20240502 14:00:40.758653 15569 raft_consensus.cc:2454] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 2.
I20240502 14:00:40.759735 15637 leader_election.cc:302] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1eb0e3a6c782469fa8110f6bb6a585cf, 8484181c4ece483c8b408817fab1a917, 8fea283177394bd28d7eacf4fb744d59; no voters: 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:40.759996 18605 raft_consensus.cc:2781] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:40.760088 18605 raft_consensus.cc:686] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 2 LEADER]: Becoming Leader. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Running, Role: LEADER
I20240502 14:00:40.760243 18605 consensus_queue.cc:237] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:40.762355 15426 catalog_manager.cc:5484] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 reported cstate change: term changed from 1 to 2, leader changed from 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) to 8484181c4ece483c8b408817fab1a917 (127.8.86.130). New cstate: current_term: 2 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:40.784919 15549 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:40.785229 15549 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:40.789590 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:40.795132 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:40.795455 17488 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:40.800215 15681 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:40.809327 15811 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:40.810837 15944 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:00:40.816937 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:00:40.824131 15944 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:00:40.825389 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:40.845249 15681 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:40.853926 15964 raft_consensus.cc:1203] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240502 14:00:40.873960 15504 consensus_peers.cc:462] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:00:40.877277 17700 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task4LMC7s/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 14:00:34.146456 17703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:34.146778 17703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:34.154042 17703 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:00:35.068694 17703 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 14:00:35.559659 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b36f1f16cb3d4b8bae9303e2976dc2e7: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:35.559854 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 278d2359eb234a2b8e569cd993a004a7: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:35.560109 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a148b6d170714c99a6024d0a3efa85c8: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:35.560175 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fcc590955678424ab1c27b0b23f75544: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:35.560227 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:35.560271 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b36f1f16cb3d4b8bae9303e2976dc2e7: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:35.560319 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6dcd44a470ab434a988eea52c3916f26: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:35.560365 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af7e16dce074d40ae6277d41e034322: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:35.560417 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 17d53f89ea7b4f918ed8bb35d44f6401: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:35.560482 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 12389884a03b40f68c532a06ec761e9c: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:35.560559 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fc947d3d4f264291a81e7e95cb84f48a: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:35.560619 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af7e16dce074d40ae6277d41e034322: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:35.560667 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 12389884a03b40f68c532a06ec761e9c: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:35.560714 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:35.560757 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b36f1f16cb3d4b8bae9303e2976dc2e7: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server 8fea283177394bd28d7eacf4fb744d59
I20240502 14:00:35.647691 17703 rebalancer_tool.cc:1074] tablet 619a8ec66ba74ecea689c17b4734253a: '82d7b7201cfc431d90baca1ad0a397cf' -> '3c3658c946994e2ba9f41b64aa327d25' move scheduled
I20240502 14:00:35.691476 17703 rebalancer_tool.cc:1074] tablet 1f562ad00ec34a8d9184671ed55d84d1: '4d0f8f311e834d81bd54b9f8e7923bcd' -> 'f688370715454d6e814a78fcf85ac5c3' move scheduled
I20240502 14:00:35.787602 17703 rebalancer_tool.cc:1074] tablet 03dd735113ab4ea6bdb696b11a4efd7f: '8484181c4ece483c8b408817fab1a917' -> '8fea283177394bd28d7eacf4fb744d59' move scheduled
I20240502 14:00:35.862064 17703 rebalancer_tool.cc:1074] tablet 02d3da05d82340178e27218c2ce56b67: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '2254b3ee3db647598d781e467584f78d' move scheduled
I20240502 14:00:35.897269 17703 rebalancer_tool.cc:1074] tablet 436c127af87c49e8b7bf9b6571f7dc30: '82d7b7201cfc431d90baca1ad0a397cf' -> '3c3658c946994e2ba9f41b64aa327d25' move scheduled
I20240502 14:00:35.955189 17703 rebalancer_tool.cc:1074] tablet b07f4db6f82e4a56accdbf1bbbc5f965: '4d0f8f311e834d81bd54b9f8e7923bcd' -> 'f688370715454d6e814a78fcf85ac5c3' move scheduled
I20240502 14:00:35.962899 17703 rebalancer_tool.cc:1084] tablet d81f14b7c1d1468196d0fe6d67fe93a7: '8484181c4ece483c8b408817fab1a917' -> '8fea283177394bd28d7eacf4fb744d59' move ignored: Illegal state: tablet d81f14b7c1d1468196d0fe6d67fe93a7: replica 8fea283177394bd28d7eacf4fb744d59 is already present
I20240502 14:00:35.963007 17703 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:00:36.315146 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b36f1f16cb3d4b8bae9303e2976dc2e7: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:36.315390 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 17d53f89ea7b4f918ed8bb35d44f6401: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:36.315487 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6dcd44a470ab434a988eea52c3916f26: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:36.315587 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af7e16dce074d40ae6277d41e034322: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:36.315640 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 278d2359eb234a2b8e569cd993a004a7: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:36.315704 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 17d53f89ea7b4f918ed8bb35d44f6401: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:36.315752 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af7e16dce074d40ae6277d41e034322: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:36.315809 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 12389884a03b40f68c532a06ec761e9c: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:36.315856 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 278d2359eb234a2b8e569cd993a004a7: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:36.315896 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 12389884a03b40f68c532a06ec761e9c: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:36.315943 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:36.316038 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fc947d3d4f264291a81e7e95cb84f48a: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:36.316092 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fcc590955678424ab1c27b0b23f75544: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:36.316140 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a148b6d170714c99a6024d0a3efa85c8: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:36.316184 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 17d53f89ea7b4f918ed8bb35d44f6401: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 8fea283177394bd28d7eacf4fb744d59
I20240502 14:00:36.362689 17703 rebalancer_tool.cc:1074] tablet dfdcf1124e8a474facc14c45e8963568: '8484181c4ece483c8b408817fab1a917' -> '8fea283177394bd28d7eacf4fb744d59' move scheduled
I20240502 14:00:36.579378 17703 rebalancer_tool.cc:1074] tablet 1c004727e7c64f33bc9492184b77fde9: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '2254b3ee3db647598d781e467584f78d' move scheduled
I20240502 14:00:36.691301 17703 rebalancer_tool.cc:1074] tablet 76f07464d9bd49a9b1b9d58f6a180cf8: '82d7b7201cfc431d90baca1ad0a397cf' -> '3c3658c946994e2ba9f41b64aa327d25' move scheduled
I20240502 14:00:36.875751 17703 rebalancer_tool.cc:1074] tablet 387bdd8ae3b34f588e2170b061a5bc05: '4d0f8f311e834d81bd54b9f8e7923bcd' -> 'f688370715454d6e814a78fcf85ac5c3' move scheduled
I20240502 14:00:37.197726 17703 rebalancer_tool.cc:1074] tablet d81f14b7c1d1468196d0fe6d67fe93a7: '8484181c4ece483c8b408817fab1a917' -> '2254b3ee3db647598d781e467584f78d' move scheduled
I20240502 14:00:37.268359 17703 rebalancer_tool.cc:1074] tablet 28e371c1a999431499e09479e8f5b44b: '4d0f8f311e834d81bd54b9f8e7923bcd' -> '8fea283177394bd28d7eacf4fb744d59' move scheduled
I20240502 14:00:37.379168 17703 rebalancer_tool.cc:1084] tablet a7981c3f46d8483caae82fbf2296e762: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '3c3658c946994e2ba9f41b64aa327d25' move ignored: Illegal state: tablet a7981c3f46d8483caae82fbf2296e762: replica 3c3658c946994e2ba9f41b64aa327d25 is already present
I20240502 14:00:37.379242 17703 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:00:38.773994 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 8484181c4ece483c8b408817fab1a917
W20240502 14:00:38.774147 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 12389884a03b40f68c532a06ec761e9c: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:38.774199 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fcc590955678424ab1c27b0b23f75544: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:38.774328 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6dcd44a470ab434a988eea52c3916f26: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 2254b3ee3db647598d781e467584f78d
I20240502 14:00:38.890945 17703 rebalancer_tool.cc:1074] tablet a7981c3f46d8483caae82fbf2296e762: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '3c3658c946994e2ba9f41b64aa327d25' move scheduled
I20240502 14:00:38.987470 17703 rebalancer_tool.cc:1074] tablet 22eef55211ca4f65979022d2f4ae829f: 'f688370715454d6e814a78fcf85ac5c3' -> '8484181c4ece483c8b408817fab1a917' move scheduled
I20240502 14:00:39.012040 17703 rebalancer_tool.cc:1074] tablet b0cd21dc897b41aca9118be6df48cc0c: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '2254b3ee3db647598d781e467584f78d' move scheduled
I20240502 14:00:39.031795 17703 rebalancer_tool.cc:1084] tablet 2886dfbec1e141198b0cf7a435fac498: '4d0f8f311e834d81bd54b9f8e7923bcd' -> '8fea283177394bd28d7eacf4fb744d59' move ignored: Illegal state: tablet 2886dfbec1e141198b0cf7a435fac498: replica 8fea283177394bd28d7eacf4fb744d59 is already present
I20240502 14:00:39.031875 17703 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:00:39.998842 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 3c3658c946994e2ba9f41b64aa327d25 to server 8484181c4ece483c8b408817fab1a917
W20240502 14:00:39.998960 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fc947d3d4f264291a81e7e95cb84f48a: could not find any suitable replica to move from server 3c3658c946994e2ba9f41b64aa327d25 to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:39.999001 17703 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 278d2359eb234a2b8e569cd993a004a7: could not find any suitable replica to move from server 8fea283177394bd28d7eacf4fb744d59 to server 8484181c4ece483c8b408817fab1a917
I20240502 14:00:40.082855 17703 rebalancer_tool.cc:1074] tablet 48fbc670ec504d26a11bf1b9fbd78981: '8484181c4ece483c8b408817fab1a917' -> '4d0f8f311e834d81bd54b9f8e7923bcd' move scheduled
I20240502 14:00:40.154671 17703 rebalancer_tool.cc:1136] tablet d81f14b7c1d1468196d0fe6d67fe93a7: '8484181c4ece483c8b408817fab1a917' -> '2254b3ee3db647598d781e467584f78d' move completed: OK
I20240502 14:00:40.212396 17703 rebalancer_tool.cc:1136] tablet 76f07464d9bd49a9b1b9d58f6a180cf8: '82d7b7201cfc431d90baca1ad0a397cf' -> '3c3658c946994e2ba9f41b64aa327d25' move completed: Incomplete: tablet 76f07464d9bd49a9b1b9d58f6a180cf8, TS 82d7b7201cfc431d90baca1ad0a397cf -> TS 3c3658c946994e2ba9f41b64aa327d25 move failed, target replica disappeared
I20240502 14:00:40.269150 17703 rebalancer_tool.cc:1136] tablet dfdcf1124e8a474facc14c45e8963568: '8484181c4ece483c8b408817fab1a917' -> '8fea283177394bd28d7eacf4fb744d59' move completed: Incomplete: tablet dfdcf1124e8a474facc14c45e8963568, TS 8484181c4ece483c8b408817fab1a917 -> TS 8fea283177394bd28d7eacf4fb744d59 move failed, target replica disappeared
I20240502 14:00:40.278635 17703 rebalancer_tool.cc:1136] tablet 1f562ad00ec34a8d9184671ed55d84d1: '4d0f8f311e834d81bd54b9f8e7923bcd' -> 'f688370715454d6e814a78fcf85ac5c3' move completed: OK
I20240502 14:00:40.307722 17703 rebalancer_tool.cc:1136] tablet 03dd735113ab4ea6bdb696b11a4efd7f: '8484181c4ece483c8b408817fab1a917' -> '8fea283177394bd28d7eacf4fb744d59' move completed: OK
I20240502 14:00:40.345340 17703 rebalancer_tool.cc:1136] tablet 619a8ec66ba74ecea689c17b4734253a: '82d7b7201cfc431d90baca1ad0a397cf' -> '3c3658c946994e2ba9f41b64aa327d25' move completed: Incomplete: tablet 619a8ec66ba74ecea689c17b4734253a, TS 82d7b7201cfc431d90baca1ad0a397cf -> TS 3c3658c946994e2ba9f41b64aa327d25 move failed, target replica disappeared
I20240502 14:00:40.359472 17703 rebalancer_tool.cc:1136] tablet a7981c3f46d8483caae82fbf2296e762: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '3c3658c946994e2ba9f41b64aa327d25' move completed: OK
I20240502 14:00:40.401460 17703 rebalancer_tool.cc:1136] tablet 28e371c1a999431499e09479e8f5b44b: '4d0f8f311e834d81bd54b9f8e7923bcd' -> '8fea283177394bd28d7eacf4fb744d59' move completed: OK
I20240502 14:00:40.425024 17703 rebalancer_tool.cc:1136] tablet 02d3da05d82340178e27218c2ce56b67: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '2254b3ee3db647598d781e467584f78d' move completed: OK
I20240502 14:00:40.445937 17703 rebalancer_tool.cc:1136] tablet 436c127af87c49e8b7bf9b6571f7dc30: '82d7b7201cfc431d90baca1ad0a397cf' -> '3c3658c946994e2ba9f41b64aa327d25' move completed: OK
I20240502 14:00:40.454687 17703 rebalancer_tool.cc:1136] tablet 1c004727e7c64f33bc9492184b77fde9: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '2254b3ee3db647598d781e467584f78d' move completed: OK
I20240502 14:00:40.468127 17703 rebalancer_tool.cc:1136] tablet b07f4db6f82e4a56accdbf1bbbc5f965: '4d0f8f311e834d81bd54b9f8e7923bcd' -> 'f688370715454d6e814a78fcf85ac5c3' move completed: OK
W20240502 14:00:40.672417 20514 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:00:40.482562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.86.133:42151 (local address 127.0.0.1:60512)
0502 14:00:40.489470 (+ 6908us) negotiation.cc:107] Waiting for socket to connect
0502 14:00:40.489486 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0502 14:00:40.489594 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:00:40.671784 (+182190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":6755,"thread_start_us":115,"threads_started":1}
I20240502 14:00:40.672869 17703 rebalancer_tool.cc:1123] tablet 387bdd8ae3b34f588e2170b061a5bc05: '4d0f8f311e834d81bd54b9f8e7923bcd' -> 'f688370715454d6e814a78fcf85ac5c3' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:00:40.688866 17703 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): unacceptable health status UNAVAILABLE
I20240502 14:00:40.888335 17698 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-task4LMC7s/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 14:00:34.077381 17704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:34.077591 17704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:34.080996 17704 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:00:35.035403 17704 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 14:00:35.627462 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 278d2359eb234a2b8e569cd993a004a7: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:35.627709 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 17d53f89ea7b4f918ed8bb35d44f6401: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:35.627830 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:35.627869 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b36f1f16cb3d4b8bae9303e2976dc2e7: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:35.627899 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6dcd44a470ab434a988eea52c3916f26: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:35.627933 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af7e16dce074d40ae6277d41e034322: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:35.637015 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a148b6d170714c99a6024d0a3efa85c8: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:35.637137 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 12389884a03b40f68c532a06ec761e9c: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:35.637184 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fcc590955678424ab1c27b0b23f75544: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:35.637235 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fc947d3d4f264291a81e7e95cb84f48a: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:35.637286 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b36f1f16cb3d4b8bae9303e2976dc2e7: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:35.637321 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af7e16dce074d40ae6277d41e034322: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:35.637357 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:35.637392 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6dcd44a470ab434a988eea52c3916f26: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 8fea283177394bd28d7eacf4fb744d59
I20240502 14:00:35.717276 17704 rebalancer_tool.cc:1074] tablet 619a8ec66ba74ecea689c17b4734253a: '82d7b7201cfc431d90baca1ad0a397cf' -> '8fea283177394bd28d7eacf4fb744d59' move scheduled
I20240502 14:00:35.733656 17704 rebalancer_tool.cc:1084] tablet 1f562ad00ec34a8d9184671ed55d84d1: '8484181c4ece483c8b408817fab1a917' -> 'f688370715454d6e814a78fcf85ac5c3' move ignored: Illegal state: tablet 1f562ad00ec34a8d9184671ed55d84d1: replica f688370715454d6e814a78fcf85ac5c3 is already present
I20240502 14:00:35.733762 17704 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:00:36.153143 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6dcd44a470ab434a988eea52c3916f26: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:36.153227 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b36f1f16cb3d4b8bae9303e2976dc2e7: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:36.153472 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fcc590955678424ab1c27b0b23f75544: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:36.153605 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b36f1f16cb3d4b8bae9303e2976dc2e7: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:36.153710 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 17d53f89ea7b4f918ed8bb35d44f6401: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:36.153774 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a148b6d170714c99a6024d0a3efa85c8: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:36.153810 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 278d2359eb234a2b8e569cd993a004a7: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:36.153872 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 17d53f89ea7b4f918ed8bb35d44f6401: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:36.153928 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 12389884a03b40f68c532a06ec761e9c: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:36.153966 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 2254b3ee3db647598d781e467584f78d
W20240502 14:00:36.154024 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b36f1f16cb3d4b8bae9303e2976dc2e7: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:36.154062 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0af7e16dce074d40ae6277d41e034322: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:36.154095 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fcc590955678424ab1c27b0b23f75544: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:36.154126 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a148b6d170714c99a6024d0a3efa85c8: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server f688370715454d6e814a78fcf85ac5c3
W20240502 14:00:36.154157 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 17d53f89ea7b4f918ed8bb35d44f6401: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:36.206969 17704 rebalancer_tool.cc:1074] tablet 22eef55211ca4f65979022d2f4ae829f: '8484181c4ece483c8b408817fab1a917' -> 'f688370715454d6e814a78fcf85ac5c3' move scheduled
I20240502 14:00:36.389916 17704 rebalancer_tool.cc:1074] tablet 2886dfbec1e141198b0cf7a435fac498: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '2254b3ee3db647598d781e467584f78d' move scheduled
I20240502 14:00:36.666675 17704 rebalancer_tool.cc:1074] tablet 03dd735113ab4ea6bdb696b11a4efd7f: '4d0f8f311e834d81bd54b9f8e7923bcd' -> '3c3658c946994e2ba9f41b64aa327d25' move scheduled
I20240502 14:00:36.788650 17704 rebalancer_tool.cc:1074] tablet 436c127af87c49e8b7bf9b6571f7dc30: '82d7b7201cfc431d90baca1ad0a397cf' -> '8fea283177394bd28d7eacf4fb744d59' move scheduled
I20240502 14:00:37.217011 17704 rebalancer_tool.cc:1074] tablet 186a946321e445b58b0097c658847ebe: '8484181c4ece483c8b408817fab1a917' -> 'f688370715454d6e814a78fcf85ac5c3' move scheduled
I20240502 14:00:37.312744 17704 rebalancer_tool.cc:1074] tablet e51db70f5f284c4da3c1669fddb41b00: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '2254b3ee3db647598d781e467584f78d' move scheduled
I20240502 14:00:37.440723 17704 rebalancer_tool.cc:1074] tablet 6e76bd0ecd3e4c168e141fb7984af3ec: '4d0f8f311e834d81bd54b9f8e7923bcd' -> '8fea283177394bd28d7eacf4fb744d59' move scheduled
I20240502 14:00:37.508496 17704 rebalancer_tool.cc:1084] tablet 9870bc797f184a6284ff6ca4f78b3826: '82d7b7201cfc431d90baca1ad0a397cf' -> '3c3658c946994e2ba9f41b64aa327d25' move ignored: Illegal state: tablet 9870bc797f184a6284ff6ca4f78b3826: replica 3c3658c946994e2ba9f41b64aa327d25 is already present
I20240502 14:00:37.508584 17704 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:00:38.825029 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 278d2359eb234a2b8e569cd993a004a7: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:38.825174 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fcc590955678424ab1c27b0b23f75544: could not find any suitable replica to move from server 82d7b7201cfc431d90baca1ad0a397cf to server 3c3658c946994e2ba9f41b64aa327d25
W20240502 14:00:38.825340 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a148b6d170714c99a6024d0a3efa85c8: could not find any suitable replica to move from server 8484181c4ece483c8b408817fab1a917 to server 8fea283177394bd28d7eacf4fb744d59
W20240502 14:00:38.825529 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6dcd44a470ab434a988eea52c3916f26: could not find any suitable replica to move from server 1eb0e3a6c782469fa8110f6bb6a585cf to server 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:38.871055 17704 rebalancer_tool.cc:1084] tablet 387bdd8ae3b34f588e2170b061a5bc05: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '' move ignored: Not found: Peer with uuid 1eb0e3a6c782469fa8110f6bb6a585cf not found in consensus config
I20240502 14:00:38.871174 17704 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:00:39.289872 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a00d0c0f593149dcb05a21e3d90dda57: could not find any suitable replica to move from server 4d0f8f311e834d81bd54b9f8e7923bcd to server 8484181c4ece483c8b408817fab1a917
W20240502 14:00:39.289983 17704 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a148b6d170714c99a6024d0a3efa85c8: could not find any suitable replica to move from server f688370715454d6e814a78fcf85ac5c3 to server 3c3658c946994e2ba9f41b64aa327d25
I20240502 14:00:39.367411 17704 rebalancer_tool.cc:1074] tablet 28e371c1a999431499e09479e8f5b44b: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '3c3658c946994e2ba9f41b64aa327d25' move scheduled
I20240502 14:00:39.437034 17704 rebalancer_tool.cc:1074] tablet dfdcf1124e8a474facc14c45e8963568: 'f688370715454d6e814a78fcf85ac5c3' -> '8484181c4ece483c8b408817fab1a917' move scheduled
I20240502 14:00:39.493551 17704 rebalancer_tool.cc:1074] tablet b0cd21dc897b41aca9118be6df48cc0c: '2254b3ee3db647598d781e467584f78d' -> 'f688370715454d6e814a78fcf85ac5c3' move scheduled
I20240502 14:00:39.658515 17704 rebalancer_tool.cc:1074] tablet bb9f100600a246708457280d91826fce: '82d7b7201cfc431d90baca1ad0a397cf' -> '4d0f8f311e834d81bd54b9f8e7923bcd' move scheduled
I20240502 14:00:39.779156 17704 rebalancer_tool.cc:1074] tablet a7981c3f46d8483caae82fbf2296e762: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '4d0f8f311e834d81bd54b9f8e7923bcd' move scheduled
I20240502 14:00:40.011329 17704 rebalancer_tool.cc:1074] tablet 1a5e33f337bd47b2a6b11b29a5eacad8: '82d7b7201cfc431d90baca1ad0a397cf' -> 'f688370715454d6e814a78fcf85ac5c3' move scheduled
I20240502 14:00:40.123956 17704 rebalancer_tool.cc:1136] tablet a7981c3f46d8483caae82fbf2296e762: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '4d0f8f311e834d81bd54b9f8e7923bcd' move completed: Incomplete: tablet a7981c3f46d8483caae82fbf2296e762, TS 1eb0e3a6c782469fa8110f6bb6a585cf -> TS 4d0f8f311e834d81bd54b9f8e7923bcd move failed, target replica disappeared
I20240502 14:00:40.240792 17704 rebalancer_tool.cc:1136] tablet e51db70f5f284c4da3c1669fddb41b00: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '2254b3ee3db647598d781e467584f78d' move completed: OK
I20240502 14:00:40.249751 17704 rebalancer_tool.cc:1123] tablet 22eef55211ca4f65979022d2f4ae829f: '8484181c4ece483c8b408817fab1a917' -> 'f688370715454d6e814a78fcf85ac5c3' move is abandoned: Illegal state: 22eef55211ca4f65979022d2f4ae829f: source replica 8484181c4ece483c8b408817fab1a917 does not have REPLACE attribute set
I20240502 14:00:40.261860 17704 rebalancer_tool.cc:1136] tablet 619a8ec66ba74ecea689c17b4734253a: '82d7b7201cfc431d90baca1ad0a397cf' -> '8fea283177394bd28d7eacf4fb744d59' move completed: OK
I20240502 14:00:40.269709 17704 rebalancer_tool.cc:1136] tablet 186a946321e445b58b0097c658847ebe: '8484181c4ece483c8b408817fab1a917' -> 'f688370715454d6e814a78fcf85ac5c3' move completed: Incomplete: tablet 186a946321e445b58b0097c658847ebe, TS 8484181c4ece483c8b408817fab1a917 -> TS f688370715454d6e814a78fcf85ac5c3 move failed, target replica disappeared
I20240502 14:00:40.278739 17704 rebalancer_tool.cc:1136] tablet 2886dfbec1e141198b0cf7a435fac498: '1eb0e3a6c782469fa8110f6bb6a585cf' -> '2254b3ee3db647598d781e467584f78d' move completed: Incomplete: tablet 2886dfbec1e141198b0cf7a435fac498, TS 1eb0e3a6c782469fa8110f6bb6a585cf -> TS 2254b3ee3db647598d781e467584f78d move failed, target replica disappeared
I20240502 14:00:40.296943 17704 rebalancer_tool.cc:1136] tablet 03dd735113ab4ea6bdb696b11a4efd7f: '4d0f8f311e834d81bd54b9f8e7923bcd' -> '3c3658c946994e2ba9f41b64aa327d25' move completed: OK
I20240502 14:00:40.338847 17704 rebalancer_tool.cc:1136] tablet 436c127af87c49e8b7bf9b6571f7dc30: '82d7b7201cfc431d90baca1ad0a397cf' -> '8fea283177394bd28d7eacf4fb744d59' move completed: OK
W20240502 14:00:40.672606 20565 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:00:40.648981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.86.133:42151 (local address 127.0.0.1:60518)
0502 14:00:40.649230 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0502 14:00:40.649241 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0502 14:00:40.649305 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:00:40.672010 (+ 22705us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":128,"threads_started":1}
I20240502 14:00:40.675575 17704 rebalancer_tool.cc:1123] tablet 6e76bd0ecd3e4c168e141fb7984af3ec: '4d0f8f311e834d81bd54b9f8e7923bcd' -> '8fea283177394bd28d7eacf4fb744d59' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:00:40.675642 17704 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): unacceptable health status UNAVAILABLE
W20240502 14:00:40.903697 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:00:40.949474 20406 raft_consensus.cc:1050] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf: attempting to promote NON_VOTER 8484181c4ece483c8b408817fab1a917 to VOTER
I20240502 14:00:40.949899 20406 consensus_queue.cc:237] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } }
I20240502 14:00:40.951586 15702 raft_consensus.cc:1261] T dfdcf1124e8a474facc14c45e8963568 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 14:00:40.952059 20406 consensus_queue.cc:1035] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240502 14:00:40.952209 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:40.952314 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:40.966094 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:41.009656 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
I20240502 14:00:41.079236 15702 raft_consensus.cc:1203] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240502 14:00:41.081135 19694 raft_consensus.cc:1050] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf: attempting to promote NON_VOTER 4d0f8f311e834d81bd54b9f8e7923bcd to VOTER
I20240502 14:00:41.081558 19694 consensus_queue.cc:237] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } }
I20240502 14:00:41.082944 15960 raft_consensus.cc:1261] T bb9f100600a246708457280d91826fce P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:41.083070 15832 raft_consensus.cc:1261] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:41.083477 20387 consensus_queue.cc:1035] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:41.083766 20678 consensus_queue.cc:1035] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 14:00:41.083922 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:41.084682 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:41.085471 20387 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } } }
I20240502 14:00:41.085903 15832 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } } }
I20240502 14:00:41.088388 15424 catalog_manager.cc:5484] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 8 to 9, 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:41.088999 15960 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: NON_VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } } }
W20240502 14:00:41.137955 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:41.194267 15569 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 2 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 14:00:41.194759 19756 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:00:41.274873 17514 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 8fea283177394bd28d7eacf4fb744d59 [term 2 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 14:00:41.275307 19756 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 14:00:41.279390 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:41.281872 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:00:41.378116 18502 raft_consensus.cc:1050] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917: attempting to promote NON_VOTER 4d0f8f311e834d81bd54b9f8e7923bcd to VOTER
I20240502 14:00:41.378571 18502 consensus_queue.cc:237] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } }
I20240502 14:00:41.379840 15960 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEARNER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:41.379916 15832 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:41.380314 19756 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:41.380430 20688 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 14:00:41.380525 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:41.382187 20688 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } } }
I20240502 14:00:41.382685 15832 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } } }
I20240502 14:00:41.383997 15960 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } } }
I20240502 14:00:41.384919 15426 catalog_manager.cc:5484] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 9 to 10, 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:00:41.398485 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:41.478684 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
I20240502 14:00:41.535849 20168 raft_consensus.cc:1050] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf: attempting to promote NON_VOTER 8484181c4ece483c8b408817fab1a917 to VOTER
I20240502 14:00:41.536280 20168 consensus_queue.cc:237] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false replace: true } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } }
I20240502 14:00:41.537840 15702 raft_consensus.cc:1261] T 02d3da05d82340178e27218c2ce56b67 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:00:41.538370 20705 consensus_queue.cc:1035] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 14:00:41.538779 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:41.538848 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:41.678681 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:41.806408 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:41.886483 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:41.895666 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22eef55211ca4f65979022d2f4ae829f: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:41.895797 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:00:42.003237 19756 raft_consensus.cc:482] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:42.003391 19756 raft_consensus.cc:504] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:42.003930 19756 leader_election.cc:288] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719), 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
I20240502 14:00:42.004274 15960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e76bd0ecd3e4c168e141fb7984af3ec" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" is_pre_election: true
I20240502 14:00:42.004369 17514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e76bd0ecd3e4c168e141fb7984af3ec" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "8fea283177394bd28d7eacf4fb744d59" is_pre_election: true
I20240502 14:00:42.004432 15960 raft_consensus.cc:2454] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 2.
I20240502 14:00:42.004581 17514 raft_consensus.cc:2454] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 2.
I20240502 14:00:42.005062 15637 leader_election.cc:302] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 8484181c4ece483c8b408817fab1a917, 8fea283177394bd28d7eacf4fb744d59; no voters:
W20240502 14:00:42.005203 15636 leader_election.cc:334] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:00:42.005240 19756 raft_consensus.cc:2781] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 14:00:42.005326 19756 raft_consensus.cc:482] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:42.005383 19756 raft_consensus.cc:3037] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:00:42.006477 19756 raft_consensus.cc:504] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:42.006918 19756 leader_election.cc:288] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 3 election: Requested vote from peers 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719), 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
I20240502 14:00:42.007187 15960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e76bd0ecd3e4c168e141fb7984af3ec" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd"
I20240502 14:00:42.007308 15960 raft_consensus.cc:3037] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:00:42.007376 17514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e76bd0ecd3e4c168e141fb7984af3ec" candidate_uuid: "8484181c4ece483c8b408817fab1a917" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "8fea283177394bd28d7eacf4fb744d59"
I20240502 14:00:42.007476 17514 raft_consensus.cc:3037] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59 [term 2 FOLLOWER]: Advancing to term 3
W20240502 14:00:42.008021 15636 leader_election.cc:334] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:00:42.008329 15960 raft_consensus.cc:2454] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 3.
I20240502 14:00:42.008399 17514 raft_consensus.cc:2454] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8484181c4ece483c8b408817fab1a917 in term 3.
I20240502 14:00:42.008729 15637 leader_election.cc:302] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd, 8484181c4ece483c8b408817fab1a917, 8fea283177394bd28d7eacf4fb744d59; no voters: f688370715454d6e814a78fcf85ac5c3
I20240502 14:00:42.008972 19756 raft_consensus.cc:2781] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:00:42.009071 19756 raft_consensus.cc:686] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 3 LEADER]: Becoming Leader. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Running, Role: LEADER
I20240502 14:00:42.009213 19756 consensus_queue.cc:237] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } }
I20240502 14:00:42.011005 15424 catalog_manager.cc:5484] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 reported cstate change: term changed from 2 to 3, leader changed from f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) to 8484181c4ece483c8b408817fab1a917 (127.8.86.130). New cstate: current_term: 3 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:42.085388 19285 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:42.085549 19285 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:42.085976 19285 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:00:42.087002 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:42.087101 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:00:42.087164 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:00:42.087291 19285 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:00:42.133925 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:42.175441 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:42.177286 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:42.233467 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:42.319605 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:42.424405 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:42.427917 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:42.427917 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:42.475757 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:42.502314 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:42.522614 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:42.550814 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:42.554208 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:00:42.572213 15960 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 3 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 14:00:42.572687 20688 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 14:00:42.574162 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:00:42.594424 17514 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8fea283177394bd28d7eacf4fb744d59 [term 3 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 14:00:42.594933 20688 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 14:00:42.617924 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:42.645005 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:42.809559 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:42.817193 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:42.899843 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:42.914058 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:42.947650 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:42.963793 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22eef55211ca4f65979022d2f4ae829f: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:42.966025 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:43.037566 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:43.092941 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:43.120252 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:43.211930 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:43.229218 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:43.414510 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:43.418325 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:43.478085 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:43.538616 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:43.558439 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:43.569727 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:43.670780 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:43.702576 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:43.710217 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:00:43.735549 19285 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:43.735709 19285 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:43.736099 19285 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:00:43.737084 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:43.737174 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:00:43.737236 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:00:43.737354 19285 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:00:43.784484 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:43.800251 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:43.800386 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:43.808055 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:43.815124 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:43.823892 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:44.006358 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:44.125090 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:44.175552 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:44.256265 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:44.356964 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:44.614226 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:44.664599 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:44.681221 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:44.725581 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:44.802969 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:44.829128 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:44.890663 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:44.949158 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:45.011271 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:45.062916 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22eef55211ca4f65979022d2f4ae829f: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:45.073135 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 115 similar messages]
W20240502 14:00:45.073788 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:00:45.084488 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 14:00:45.091711 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:45.121534 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240502 14:00:45.154541 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:45.155241 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:45.171983 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:45.177098 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:00:45.203416 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:45.210247 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:45.212309 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:45.215751 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:45.314005 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:45.332803 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:00:45.333513 19285 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:45.333603 19285 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:45.333920 19285 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:00:45.334856 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:45.334941 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:00:45.335032 15897 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:00:45.335177 19285 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:00:45.350536 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:45.361730 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:45.372942 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:45.582547 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:45.643855 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:45.712682 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:45.756172 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:45.821672 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:45.853099 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:45.896082 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:45.927275 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:45.929813 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:45.986012 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:46.121631 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:46.130856 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:46.137142 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:46.180160 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:46.291997 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:46.318792 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:46.323146 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:46.333410 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:46.391274 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:46.395370 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:46.396179 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:46.693972 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:46.715924 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:46.778626 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:46.968181 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:47.014820 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:47.047178 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:47.114580 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:47.212810 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:47.267668 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:47.283315 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:47.446944 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:47.467339 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:47.497884 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:00:47.557734 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:47.686009 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:47.703457 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:47.752583 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:47.774026 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:47.776535 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:47.798332 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:47.827399 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:47.827934 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:47.911386 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:47.929870 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:47.948146 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:47.992965 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:48.019191 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:48.104207 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:48.142344 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:48.250607 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:48.276659 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:48.276801 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:48.286720 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:48.332455 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:48.427230 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:48.529531 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:48.556008 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:48.559270 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:48.609412 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:48.650946 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:48.773419 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:48.776475 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:48.790151 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:48.849642 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:48.889178 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:48.893873 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:48.939653 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:49.121071 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:49.181653 15414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240502 14:00:49.182504 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22eef55211ca4f65979022d2f4ae829f: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:49.191905 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:49.388371 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:49.475468 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:49.480572 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:00:49.531862 20868 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:49.531972 20868 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:49.532298 20868 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:00:49.533123 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:49.533228 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:00:49.533285 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:00:49.533438 20868 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:00:49.605659 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:49.608515 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:49.629240 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:49.841107 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:49.843528 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:49.865552 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:49.891541 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:49.937076 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:00:50.082897 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:00:50.094686 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:50.096663 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:50.102396 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240502 14:00:50.102963 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:50.127249 15505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 14:00:50.139051 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:50.147650 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:50.184823 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:00:50.185883 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:50.377498 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:50.411702 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:50.416162 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:50.438114 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:50.456929 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:50.461350 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:50.571300 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:50.679371 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:50.691303 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:50.732764 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:50.735682 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:50.757239 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:50.798539 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:50.851707 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:50.904794 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:50.940650 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:50.944453 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:50.972863 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:51.004081 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:51.020538 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:51.039144 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:51.118359 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:51.249512 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:51.291249 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:51.311081 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:51.325649 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:51.339771 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:51.368242 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:51.426411 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:51.716848 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:51.826532 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:51.926625 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:51.927371 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:51.983368 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:52.170745 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:52.198333 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:52.318651 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:52.427050 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:52.448237 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:00:52.479089 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:52.496815 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:52.557637 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:52.661103 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:52.695164 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:52.751555 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:52.807204 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:52.844604 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:52.884919 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:52.885993 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:52.911327 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:52.932027 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:52.961555 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:52.975999 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:53.088236 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:53.137398 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:53.158690 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:53.211735 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:53.238705 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:53.244094 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:53.314188 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:53.333411 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:53.352931 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:53.369163 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:53.384212 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:53.411258 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:53.499583 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:53.526180 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:53.533344 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:53.563148 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:53.735205 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:53.836212 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:53.896133 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:53.917333 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:53.961373 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:53.973074 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:53.990485 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:54.000267 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:54.178067 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:54.282606 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:54.289124 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:54.335036 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:54.427556 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:54.587456 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:54.596133 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:54.722069 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:54.814327 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:54.948931 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:54.949846 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:54.952042 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:54.962546 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:55.022469 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:55.038045 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:55.101781 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 14:00:55.127880 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:55.199147 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 14:00:55.199925 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:00:55.241781 21022 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:55.241899 21022 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:00:55.242312 21022 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:00:55.242736 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:00:55.243346 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:55.243413 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:00:55.243505 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:00:55.243623 21022 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:00:55.292153 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:55.315117 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240502 14:00:55.315735 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:55.340566 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:55.372961 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:55.395002 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:55.414497 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:55.483227 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:55.553719 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:55.593489 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:55.731921 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:55.754876 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:55.813081 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:55.818801 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:55.838248 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:55.857149 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:55.885246 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:55.913101 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:55.949831 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:55.992465 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:56.108865 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:56.117374 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:56.120898 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:56.221489 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:56.290670 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:56.296779 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:56.381319 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:56.426658 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:56.484398 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:56.509629 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:56.532511 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:56.606019 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:56.724838 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:56.826602 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:56.851924 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:56.855901 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:56.896620 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:57.124744 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:57.230020 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:57.306195 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:57.306692 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:57.387873 15414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:57.388512 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22eef55211ca4f65979022d2f4ae829f: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:57.391220 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:57.506390 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:57.557171 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:57.619534 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:57.664777 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:57.688037 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:57.692379 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:57.696105 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:57.739048 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:57.757984 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:57.777046 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:57.795887 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:57.809042 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:57.911202 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:57.963294 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:57.964290 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:58.053516 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:58.225327 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:00:58.247946 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:58.279167 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:58.296567 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:58.301452 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:58.343328 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:58.355939 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:58.385756 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:58.405299 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:58.425618 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:58.436398 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:58.466006 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:58.614943 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:58.727559 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:58.728318 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:58.839869 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:58.841543 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:58.842996 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:58.881731 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:58.944269 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:58.982734 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:59.117558 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:59.120826 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:59.272518 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:59.294314 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:00:59.300827 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:59.400250 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:00:59.428776 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:00:59.669057 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:00:59.710350 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:00:59.789654 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:00:59.985880 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:59.988480 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:00.063158 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:00.078727 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:00.105471 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:00.116997 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:00.189770 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:01:00.190975 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:00.221516 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 14:01:00.222132 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:00.233485 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:00.246615 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:00.247339 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 14:01:00.247829 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:00.295981 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:00.373183 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240502 14:01:00.415020 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:00.434362 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:00.460992 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:00.492610 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:00.509091 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:00.551508 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:01:00.826575 21174 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:00.826743 21174 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:01:00.827207 21174 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:01:00.827760 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:00.828492 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:00.828581 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:01:00.828667 15897 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:01:00.828799 21174 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:00.853046 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:00.857679 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:00.867331 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:00.880170 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:00.908713 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:00.912592 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:00.920845 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:00.921689 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:00.926985 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:01.003163 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:01.004596 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:01.073112 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:01.134495 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:01.167196 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:01.306839 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:01.346766 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:01.351898 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:01.391276 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:01.391283 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:01.428668 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:01.712632 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:01.733043 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:01.745838 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:01.801550 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:01.855543 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:01.899492 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:01.914947 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:02.255854 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:02.272171 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:02.326217 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:02.375151 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:02.412860 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:02.414790 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:02.575650 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:02.615505 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:02.660789 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:02.663059 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:02.672742 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:02.690654 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:02.764567 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:02.776593 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:02.807761 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:02.845564 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:02.886111 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:02.965291 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:03.016597 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:03.186945 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:03.203253 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:03.283179 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:03.350672 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:03.403681 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:03.410912 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:03.416141 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:03.424289 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:03.449703 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:03.480991 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:03.516753 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:03.557338 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:03.568774 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:03.586375 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:03.687395 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:03.752350 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:03.814172 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:03.866039 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:03.891471 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:03.923573 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:04.004774 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:04.011512 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:04.087334 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:04.212616 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:04.310379 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:04.336899 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:04.487761 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:04.535895 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:04.670564 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:04.675668 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:04.812934 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:04.919129 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:04.943974 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:04.989408 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:05.007325 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:05.028211 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:05.048771 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:05.069869 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:05.110505 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:05.267855 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:05.289445 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 14:01:05.310452 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:01:05.330766 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:01:05.343591 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:05.381426 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:05.391530 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:05.395210 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:05.403208 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:05.413169 15505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:01:05.435981 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:05.502676 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:05.624042 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:05.726778 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:05.801198 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:05.809166 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:05.823412 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:05.840541 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:05.924204 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:05.960076 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:05.978942 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:05.979167 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:06.007401 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:06.036526 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:06.092545 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:06.094888 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:06.097856 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:06.159302 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:06.316198 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:06.338277 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:06.354038 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:06.354391 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:06.443868 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:06.468129 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:06.608342 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:06.623418 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:06.652343 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:06.714948 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:06.730257 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:06.909958 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:07.205750 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:07.222541 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:07.231139 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:07.329650 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:07.331415 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:07.393622 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:07.432706 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:07.543745 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:07.550673 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:07.558996 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:07.588438 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:07.588716 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:07.645221 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:07.750478 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:07.824061 21388 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:07.824200 21388 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:01:07.824651 21388 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:01:07.825564 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:07.825599 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:07.825915 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:01:07.825975 15897 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:01:07.826123 21388 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:07.831204 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:07.837451 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:07.861652 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:07.882597 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:07.940104 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:08.135638 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:08.170902 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:08.209564 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:08.252764 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:08.287948 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:08.404415 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:08.453366 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:08.461314 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:08.489090 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:08.500419 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:08.533315 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:08.557505 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:08.573575 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:08.651240 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:08.680895 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:08.725039 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:08.822535 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:08.904937 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:08.928700 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:08.955794 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:09.010957 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:09.019320 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:09.092165 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:09.145982 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:09.188724 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:09.221107 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:09.417062 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:09.468173 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:09.581024 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:09.582386 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:09.614341 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:09.663573 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:09.759228 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:09.867622 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:10.006139 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:10.015167 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:10.030030 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:10.046577 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:10.108289 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:10.174249 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:10.257359 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:10.268477 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:10.274148 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:10.287675 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:10.314036 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:01:10.344571 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:10.418223 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 14:01:10.429502 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 14:01:10.441371 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:01:10.499831 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:10.563294 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:10.572631 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:10.621721 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:10.650866 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:01:10.804088 21388 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:10.804261 21388 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:01:10.804749 21388 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:01:10.805815 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:10.805905 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:01:10.805966 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:01:10.806114 21388 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:10.841287 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:10.868444 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:10.885077 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:11.003787 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:11.023823 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:11.045469 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:11.055223 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:11.078042 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:11.127692 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:11.161243 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:11.210182 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:11.236353 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:11.297935 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:11.327683 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:11.399775 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:11.430277 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:11.433243 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:11.521574 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:11.540439 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:11.564990 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:11.585038 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:11.656812 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:11.692445 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:11.894348 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:11.999153 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:12.029229 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:12.056717 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:12.130259 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:12.281792 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:12.353497 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:01:12.356688 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:12.488809 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:12.513396 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:12.550472 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:12.595397 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:12.633155 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:12.667675 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:12.673017 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:12.713228 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:12.829362 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:12.854152 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:12.855525 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:12.899137 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:13.021073 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:13.084354 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:13.097548 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:13.164041 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:13.286527 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:13.297557 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:13.308529 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:13.319268 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:13.338613 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:13.421115 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:13.512652 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:13.518219 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:13.547102 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:13.650858 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:13.697857 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:13.745183 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:13.780670 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:13.801396 15414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:13.801582 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:13.802134 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:13.802202 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22eef55211ca4f65979022d2f4ae829f: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:13.818465 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:13.841068 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:13.870070 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:13.941704 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:14.042222 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:01:14.042376 21562 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:14.042518 21562 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:01:14.042966 21562 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:01:14.043980 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:14.044131 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:01:14.044212 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:01:14.044346 21562 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:14.068102 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:14.169602 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:14.178653 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:14.270036 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:14.302181 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:14.338896 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:14.420231 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:14.460595 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:14.532440 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:14.656613 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:14.729391 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:14.770562 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:01:14.873857 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:14.977499 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:15.003301 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:15.060544 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:15.062829 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:15.081641 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:15.199827 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:15.209475 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:15.217465 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:15.313396 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:15.326187 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:01:15.403030 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:15.410794 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:15.444679 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 14:01:15.465806 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240502 14:01:15.470906 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:01:15.471513 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:15.480293 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:15.599402 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:15.642894 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:15.720460 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:15.755883 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:15.847793 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:15.945102 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:16.013048 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:16.025803 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:16.027246 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:16.096468 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:16.135792 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:16.140916 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:16.192945 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:16.196269 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:16.259436 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:16.312520 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:16.350100 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:16.377848 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:16.410324 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:16.455691 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:16.489459 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:16.491281 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:16.498338 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:16.529556 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:16.540993 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:16.613160 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:16.771319 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:16.906967 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:16.918449 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:16.922713 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:16.973657 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:17.003126 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:17.136780 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:01:17.143600 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:17.391319 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:17.392319 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:17.427870 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:17.446482 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:17.484694 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:17.504433 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:17.603550 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:17.680341 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:17.696055 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:17.729532 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:17.865784 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:17.976327 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:18.030696 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:18.032275 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:18.044497 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:18.059804 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:18.144773 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:18.261804 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:18.367707 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:18.412384 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:18.431979 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:18.500054 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:18.547574 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:18.559870 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:18.589609 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:18.598307 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:18.704882 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:18.711112 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:18.804747 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:18.830219 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:18.924340 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:19.012082 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:19.019026 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:19.020687 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:19.041510 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:19.055780 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:19.095664 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:19.096227 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:19.209111 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:19.213248 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:19.313434 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:19.389197 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:19.486729 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:19.558955 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:19.565935 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:19.769670 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:19.793797 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:19.800472 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:01:19.804929 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:19.828122 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:19.840713 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:19.883973 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:19.902534 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:20.086091 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:20.107158 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:20.129786 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:20.178315 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:20.261322 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:20.281108 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:20.327992 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:20.348187 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:01:20.446692 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 14:01:20.471343 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:20.489708 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:01:20.500465 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 14:01:20.676679 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:20.720383 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:20.725791 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:20.802888 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:20.838569 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:20.945972 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:21.019629 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:21.037015 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:21.047477 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:21.090927 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:21.098773 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:21.099161 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:21.143935 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:21.212613 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:21.220952 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:21.276170 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:21.279826 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:21.281342 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:21.326897 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:21.347474 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:21.386246 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:21.455204 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:21.621973 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:21.635866 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:21.653180 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:21.672302 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:21.794363 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:21.873162 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:21.894310 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:21.961136 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:22.002238 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:22.128180 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:22.238653 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:22.241945 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:01:22.282935 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:22.295972 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:22.366273 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:22.458896 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:22.465268 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:22.514828 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:22.515030 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:22.710857 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:22.752547 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:22.784212 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:22.827633 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:22.852108 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:22.964644 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:23.076838 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:23.221691 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:23.243057 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:23.285874 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:23.305588 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:23.377327 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:23.389483 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:23.428261 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:23.530843 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:23.557511 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:23.615171 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:23.680703 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:23.684376 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:23.698362 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:23.716611 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:23.726456 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:23.815455 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:23.870462 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:23.938786 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:23.976485 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:24.037256 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:24.039155 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:24.045787 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:24.126703 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:24.140278 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:24.162310 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:24.185259 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:24.239029 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:24.306605 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:24.323650 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:24.399533 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:24.532197 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:24.575178 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:24.690042 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:24.705054 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:24.714427 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:24.937206 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:01:24.959187 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:24.978127 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:25.044636 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:25.059926 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:25.128603 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:25.133806 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:25.175874 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:25.220674 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:25.274801 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:25.274984 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:25.366506 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:01:25.455999 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:25.526019 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 14:01:25.559778 15505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:01:25.593787 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:25.627492 15898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:01:25.628183 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:25.755615 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:25.857435 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:25.937929 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:25.957118 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:26.038621 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:26.048184 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:26.059684 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:26.117779 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:26.151949 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:26.152141 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:26.172133 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:26.187681 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:26.199206 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:26.232147 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:26.244992 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:26.328922 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:26.440614 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:26.489599 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:26.552289 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:26.562781 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:26.609907 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:26.614271 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:26.649991 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:26.676142 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:26.813020 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:26.845510 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:26.888116 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:26.903857 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:27.023957 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:27.092677 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:27.197435 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:27.222151 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:27.233518 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:27.257678 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:27.276923 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:27.328402 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:27.428709 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:27.554446 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:01:27.634521 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:27.672420 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:27.726362 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:27.743640 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:27.766870 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:27.772367 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:27.775485 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:27.922994 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:28.108183 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:28.232817 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:28.326331 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:28.344033 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:28.534334 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:28.576198 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:28.591742 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:28.661796 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:28.667474 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:28.699203 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:28.706306 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:28.717376 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:28.765173 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:28.776712 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:28.782078 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:28.798158 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:28.875775 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:28.909253 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:29.005617 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:29.041471 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:29.078441 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:29.131619 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:29.158998 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:29.186650 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:29.265226 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:29.269331 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:29.298226 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:29.301313 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:29.457419 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:29.490502 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:29.524355 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:29.587360 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:29.646170 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:29.710309 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:29.787453 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:29.801365 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:29.813056 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:29.892689 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:29.951542 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:01:30.031306 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:30.037017 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:30.178064 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:30.247076 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:30.254032 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:30.258131 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:30.346827 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:30.351945 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:30.368609 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:30.380787 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240502 14:01:30.562927 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 14:01:30.589834 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 14:01:30.590564 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:30.711694 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:01:30.869669 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:30.884020 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:30.953410 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:30.959374 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:30.990986 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:31.009984 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:31.029037 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:31.131455 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:31.136138 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:31.152380 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:31.261366 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:31.283104 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:31.298017 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:31.317037 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:31.356814 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:31.382849 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:31.411926 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:31.414055 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:31.465899 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:31.543455 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:31.575644 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:31.625981 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:31.648020 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:31.690783 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:31.722476 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:31.731366 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:31.764863 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:32.023424 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:32.044554 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:32.135682 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:32.217765 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:32.251243 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:32.262480 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:32.309532 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:32.320165 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:32.421336 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:32.435623 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:01:32.612053 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:32.642619 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:32.651247 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:32.662017 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:32.763535 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:32.765827 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:32.781534 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
I20240502 14:01:32.791230 22014 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:32.791380 22014 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:01:32.791779 22014 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:01:32.792948 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:32.793067 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:01:32.793155 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:01:32.793295 22014 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:32.817003 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:32.925894 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:33.045538 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:33.147008 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:33.213649 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:33.247236 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:33.248513 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:33.302760 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:33.524705 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:33.572552 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:33.574223 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:33.579262 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:33.598188 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:33.619064 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:33.665199 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:33.682130 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:33.689419 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:33.692605 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:33.709992 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:33.845623 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:33.860009 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:33.992892 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:33.995530 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:34.079838 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:34.095947 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:34.136379 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:34.159518 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:34.175779 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:34.185643 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:34.206570 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:34.319280 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:34.456274 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:34.602066 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:34.613296 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:34.642782 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:34.645843 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:34.784787 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:34.823360 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:34.827905 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:34.891583 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:01:34.904336 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:35.087324 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:35.215186 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:35.263996 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:35.272913 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:35.285494 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:35.385077 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:35.387676 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:01:35.406519 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:35.503525 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:35.539002 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:35.576862 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:35.579216 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240502 14:01:35.579816 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:35.639174 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:01:35.653751 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:35.661993 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:35.702463 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:35.778262 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:01:35.798238 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:35.883423 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:35.888298 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:35.955211 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:35.967159 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:36.075488 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:36.084789 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:36.108462 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:36.145479 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:36.196903 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:36.211539 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:36.212770 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:36.219648 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:36.445552 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:36.523510 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:36.623914 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:36.626052 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:36.708123 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:36.725337 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:36.736315 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:36.775663 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:36.775689 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:36.911689 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:36.954859 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:36.958140 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:36.958137 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:37.026373 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:37.092063 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:37.210150 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:37.245103 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:37.329605 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:37.336205 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:01:37.378448 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:37.393747 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:37.659353 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:37.706651 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:37.718160 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:37.762903 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:37.801761 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:37.810076 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:37.859723 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:37.989820 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:38.057269 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:38.129375 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:38.158177 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:38.172209 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:38.172372 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:38.223048 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:38.244870 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:38.282328 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:38.480052 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:38.481906 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:38.490911 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:38.569190 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:38.642868 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:38.649849 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:38.652541 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:38.754161 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:38.756784 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:38.771622 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:38.886737 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:38.948676 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:39.127110 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:39.165038 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:39.194671 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:39.288702 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:39.301841 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:39.315851 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:39.315994 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:39.323022 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:39.337895 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:39.382580 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:39.393155 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:39.448657 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:39.474275 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:39.573139 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:39.587487 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:39.592094 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:39.751890 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:39.795395 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:01:39.883847 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:40.092630 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:40.207408 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:40.209456 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:40.211522 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:40.319638 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:40.382704 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:40.387248 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:40.397117 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:01:40.452889 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:40.522815 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:40.536855 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:40.570425 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:40.601984 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 14:01:40.612583 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:40.614940 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:40.623636 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:40.626958 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:40.712774 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:01:40.713485 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:40.929848 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:40.952327 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:41.010560 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:41.017127 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 14:01:41.058416 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:41.093983 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:41.094652 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:41.109378 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:41.112994 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:41.235702 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:41.270560 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:41.414798 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:41.449544 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:41.531263 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:41.561859 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:41.574064 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:41.576889 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:41.651690 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:41.688530 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:41.700799 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:41.731154 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:41.741027 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:41.826571 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:41.916615 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:41.967159 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:42.004938 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:42.027765 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:42.115542 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:42.116734 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:42.169760 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:42.241694 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:42.317849 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:01:42.440887 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:42.497579 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:42.658404 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:42.713804 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:42.718361 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:42.745381 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:42.749302 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:42.855440 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:42.870831 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:42.886854 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:42.984380 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:43.091732 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:43.103365 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:43.107581 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:43.154022 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:43.210840 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:43.250427 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:43.279028 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:43.382352 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:43.418262 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:43.507126 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:43.513849 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:43.568138 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:43.651976 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:43.709803 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:43.710366 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:43.790728 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:43.811964 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:43.912961 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:43.965246 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:44.117823 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:44.126564 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:44.147737 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:44.169313 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:44.184065 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:44.190338 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:44.204913 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:44.361932 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:44.372673 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:44.500830 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:44.521000 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:44.585379 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:44.588478 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:44.588459 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:44.609514 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:44.691684 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:01:44.706243 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:44.809535 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:44.879984 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:45.074599 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:45.130707 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:45.163805 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:45.229559 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:45.270999 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:45.297214 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:45.402137 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240502 14:01:45.430934 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:45.478585 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:45.483552 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:45.529239 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:45.608944 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:45.675062 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:01:45.712100 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:45.717469 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 14:01:45.718322 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:45.719151 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:45.747314 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:45.771939 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:45.848253 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:45.850904 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:45.855295 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:46.009191 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:46.066352 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:01:46.134806 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:46.147944 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:46.225032 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:46.273805 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:46.292814 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:46.300352 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:46.348642 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:46.363103 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:46.485309 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:46.579145 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:46.594786 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:46.595099 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:46.612191 15414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:46.613058 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22eef55211ca4f65979022d2f4ae829f: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:46.641170 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:46.654095 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:01:46.668025 22302 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:46.668200 22302 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:01:46.668685 22302 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:01:46.669842 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:46.669991 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:01:46.670070 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:01:46.670204 22302 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:46.673444 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:46.700155 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:46.786103 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:46.808270 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:46.880125 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:46.886914 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:46.976708 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:47.043828 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:47.074492 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:47.213861 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:47.244450 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:01:47.268570 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:47.391204 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:47.409158 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:47.462853 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:47.630968 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:47.691093 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:47.782388 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:47.855013 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:47.904551 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:47.960079 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:47.981948 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:48.013877 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:48.032714 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:48.114840 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:48.188503 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:48.247915 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:48.266479 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:48.297238 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:48.398212 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:48.418854 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:48.515113 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:48.581498 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:48.622917 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:48.640347 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:48.668701 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:48.733002 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:48.741251 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:48.806951 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:48.811381 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:48.829545 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:48.858971 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:48.872942 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:48.990262 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:49.004413 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:49.036423 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:49.076526 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:49.114140 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:49.194741 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:49.309335 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:49.358057 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:49.429816 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:49.453167 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:49.473202 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:49.478483 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:49.513911 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:49.528730 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:49.535006 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:01:49.697432 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:49.866052 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:49.879182 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:49.912031 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:49.960758 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:50.176245 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:50.189554 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:50.254979 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:50.297860 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:50.355033 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:50.373567 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:50.382726 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:50.390473 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:50.415549 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 14:01:50.563009 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:50.581849 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:50.703430 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:50.709846 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240502 14:01:50.719033 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:50.722364 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:50.755295 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 14:01:50.765277 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:50.772624 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:50.823487 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:50.861428 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:50.944715 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:51.048085 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:51.109027 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:51.170841 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:51.204310 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:51.205705 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:51.231659 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:51.284018 15898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240502 14:01:51.284683 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:51.325958 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:51.351155 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:51.352691 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:51.381877 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:51.475081 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:51.535802 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:51.540483 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:51.597819 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:51.639663 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:51.648674 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:51.693332 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:51.734474 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:51.744175 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:51.877405 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:51.905623 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:52.168088 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:52.178434 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:52.197621 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:01:52.253620 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:52.307937 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:52.332646 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:52.336432 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:52.338255 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:52.574118 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:52.627982 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:52.642565 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:52.717976 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:52.827173 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:52.933809 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:52.959251 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:53.026473 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:53.127526 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:53.161450 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:53.277681 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:53.323040 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:53.332078 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:53.376206 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:53.381624 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:53.403656 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:53.405697 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:53.467231 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:53.520207 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:53.523020 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:53.554394 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:53.689188 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:53.864625 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:53.899272 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:53.915174 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:53.926323 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:53.959285 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:53.970679 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:53.998375 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:54.034497 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:54.084875 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:54.163583 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:54.197273 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:54.220669 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:54.285653 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:54.344064 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:54.401490 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:54.479374 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:54.567787 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:54.583731 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:54.606354 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:54.643808 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:54.686820 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:54.842943 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:54.860641 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:01:54.945006 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:54.997069 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:55.083613 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:55.100744 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:55.254348 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:55.262269 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:55.430114 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240502 14:01:55.481537 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:55.482065 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:55.516588 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:55.516971 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:55.524485 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:55.584244 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:55.620002 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:55.649394 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:55.743045 15505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:01:55.743813 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:55.771401 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 14:01:55.795379 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:55.836367 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:55.921653 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:55.922386 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:55.963290 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:55.973457 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:56.026059 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:56.026129 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:56.108605 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:56.133872 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:56.257946 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:56.294034 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:56.312623 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:56.332957 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:56.389540 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:56.394695 15898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 14:01:56.395390 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:56.429608 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:56.461937 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:56.537825 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:56.599740 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:56.667691 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:56.780825 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:56.800849 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:56.802860 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:56.988197 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:57.002490 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:57.006022 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:57.033345 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:57.074903 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:57.160313 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:57.230059 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:01:57.280956 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:57.322575 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:57.390846 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:57.398631 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:57.440724 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:57.535687 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:57.619846 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:57.737149 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:01:57.907155 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:57.933070 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:57.967633 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:58.004561 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:58.069415 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:58.094002 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:58.127754 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:58.207615 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:58.308146 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:58.332589 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:58.338002 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:01:58.338416 22533 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:58.338533 22533 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:01:58.338852 22533 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:01:58.339699 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:58.339843 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:01:58.339927 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:01:58.340051 22533 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:58.351409 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:58.374531 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:58.480690 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:58.583897 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:58.600689 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:01:58.610314 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:01:58.643308 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:58.669915 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:58.714670 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:58.742491 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:58.847092 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:58.878613 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:58.888044 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:58.900629 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:01:58.970459 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:59.065182 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:59.111197 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:01:59.112850 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:59.181149 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:59.288388 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:59.451167 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:59.487761 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:59.513727 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:59.514396 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:01:59.590049 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:01:59.611560 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:59.662112 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:59.727555 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:01:59.776216 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:01:59.794116 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:01:59.807065 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:01:59.810034 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:01:59.812724 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:00.069334 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:00.113615 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:00.249316 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:00.250666 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:00.355645 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:02:00.366147 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:00.424477 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:00.486505 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:00.514990 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:02:00.567669 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:00.628943 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:00.642781 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:00.650097 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:00.672065 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:00.806475 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:02:00.821867 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 14:02:00.944150 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:00.965458 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:00.999996 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:01.002769 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:01.003667 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:01.034377 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:01.065765 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:01.164773 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:01.170994 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:01.199187 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:01.203574 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:01.208557 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:01.237882 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:01.383291 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:01.428678 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 14:02:01.429291 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:01.478675 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:01.548874 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:01.742264 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:01.745155 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:01.766216 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:01.850499 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:01.868681 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:02.005889 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:02.031476 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:02.059294 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:02.126838 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:02.203517 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:02.226485 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:02.273603 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:02.289503 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:02.357868 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:02.375595 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:02.434648 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:02.464608 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:02.470144 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:02.526355 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:02.699470 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:02.773329 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:02.778923 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:02.799654 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:02.860198 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:02.871819 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:02.961351 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:03.060688 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:03.111532 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:03.289604 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:03.292420 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:03.349519 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:03.403391 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:03.429620 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:03.444906 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:03.461207 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:03.482592 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:03.491292 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:03.561785 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:03.569551 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:03.676625 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:03.694805 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:03.791549 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:03.805058 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:03.825659 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:03.871711 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:03.902102 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:04.022172 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:04.057952 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:04.126260 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:04.241637 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:04.310346 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:04.378350 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:04.386152 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:04.400182 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:04.516206 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:04.549655 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:04.630200 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:04.700476 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:04.725291 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:04.834424 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:04.838999 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:04.983680 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:05.058493 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:05.077494 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:05.078547 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:05.119163 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:05.126657 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:05.130231 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:05.183679 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:05.235178 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:05.322352 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:05.331295 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:05.381915 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:05.519279 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:02:05.549599 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:05.574658 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:05.657461 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:05.742002 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:05.831583 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 14:02:05.834396 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:05.878917 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:05.891618 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 14:02:05.892215 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:05.912973 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:05.931826 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:05.939877 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:05.948299 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:05.972749 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:06.056574 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:06.165549 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:06.215055 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:06.278864 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:06.309986 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:06.351737 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:06.412395 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:06.444905 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:02:06.474676 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:06.511044 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:06.548732 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:06.589083 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:06.648093 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:06.648846 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:06.680083 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:06.699718 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:06.842852 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:07.000674 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:07.077425 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:07.102875 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:07.159477 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:07.313781 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:07.337803 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:07.340698 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:07.348263 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:07.368700 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:07.408092 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:07.557121 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:07.571900 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:07.587863 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:07.607573 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:07.638111 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:07.741883 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:07.769809 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:07.778476 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:07.834741 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:08.047639 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:08.066148 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:08.131057 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:08.140111 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:08.185117 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:08.198810 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:08.228117 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:08.278079 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:08.437402 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:08.469383 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:08.508908 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:08.561259 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:08.643921 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:08.669409 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:08.722982 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:08.764911 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:08.775919 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:08.786882 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:08.796602 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:08.935165 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:09.075259 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:09.083140 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:09.116564 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:09.239938 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:09.243065 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:09.259208 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:09.268393 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:09.298619 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:09.325620 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:09.439633 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:09.627677 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:09.676105 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:09.703477 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:09.718345 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:09.898394 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:09.901515 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:09.912132 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:09.997882 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:10.033030 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:10.068176 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:10.089694 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:10.164743 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:10.170457 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:10.171731 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:10.198961 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:10.276301 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:10.289155 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:10.400933 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:10.556907 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240502 14:02:10.624236 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:10.718103 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:10.772966 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:10.780812 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:10.808363 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:10.813707 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:10.819562 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:10.885141 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:10.885515 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:10.902787 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:02:11.012497 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240502 14:02:11.077690 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:11.111064 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:11.157403 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:11.210258 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:11.251600 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:11.313730 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:11.320089 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:11.329522 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:11.340407 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:11.396811 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:11.437088 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:11.467756 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:11.583439 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:11.703599 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:11.746656 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:11.747609 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:11.777988 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:02:11.778682 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:11.839890 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:11.854796 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:11.912425 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:12.120105 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:12.127086 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:12.233158 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:12.320150 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:12.326057 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:12.402850 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:12.418567 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:12.471836 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:12.520149 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:12.523276 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:12.551807 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:12.587401 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:12.731303 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:12.741389 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:12.761626 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:12.761736 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:12.870704 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:12.917455 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:12.991191 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:13.078421 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:13.177309 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:13.211220 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:13.293670 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:13.310060 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:13.317762 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:13.323947 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:13.438405 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:13.457475 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:13.457888 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:13.498960 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:13.511734 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:13.571309 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:13.687821 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:13.760228 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:13.843405 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:13.860610 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:13.868391 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:13.937469 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:13.981328 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:14.015437 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:14.056687 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:14.090598 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:14.197201 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:14.213856 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:14.229260 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:14.247745 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:14.312198 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:14.463083 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:14.596714 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:14.616983 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:14.739792 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:14.795121 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:14.832340 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:14.923839 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:14.926677 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:15.111109 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:15.143149 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:15.145979 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:15.149626 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:15.173840 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:15.261262 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:15.312922 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:15.462553 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:15.511389 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:15.544842 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:02:15.623443 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:02:15.625546 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:15.655344 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:15.661868 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:15.713238 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:15.735970 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:15.767155 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:15.774039 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:15.797818 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:15.826381 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:15.846987 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:15.882905 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:15.949939 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:15.994633 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 14:02:15.995254 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:16.028777 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:02:16.029479 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:16.047530 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:16.076412 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:16.178112 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:16.198927 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:16.211823 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:16.360854 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:16.375161 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:16.393735 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:16.453028 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:16.550276 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:16.583700 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:16.621064 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:16.627660 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:16.675668 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:16.782768 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:16.824424 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:16.930282 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 14:02:17.006688 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:17.174400 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:17.251932 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:17.258917 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:17.320355 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:17.387529 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:17.414860 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:17.427670 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:17.455477 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:17.532373 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:17.683698 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:17.716569 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:17.777485 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:17.799556 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:17.806885 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:17.808777 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:17.930523 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:18.094074 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:02:18.146054 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:18.191179 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:02:18.224115 23279 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:02:18.224246 23279 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:02:18.224646 23279 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:02:18.225605 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:18.225739 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:02:18.225816 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:02:18.225939 23279 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:02:18.251742 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:18.255931 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:18.275589 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:18.312533 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:18.353725 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:18.376101 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:18.397675 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:18.464191 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:18.466732 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:18.498519 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:18.520888 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:18.543409 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:18.543545 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:18.551671 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:18.588941 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:18.690104 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:18.743211 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:18.758524 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:18.801043 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:18.846383 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:18.898370 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:18.988391 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:19.027858 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:19.106022 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:19.115862 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:19.391582 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:19.417383 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:19.463169 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:19.475422 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:19.520167 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:19.684918 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:19.818094 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:19.851462 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:19.865748 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:20.007550 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:20.020771 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:20.034619 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:20.068171 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:20.128495 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:20.163751 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:20.243472 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:20.311496 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:20.361202 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:20.411532 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:20.540154 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:20.554749 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:20.632894 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:02:20.674688 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:20.696194 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:20.757050 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:20.762722 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:20.787279 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:02:20.799000 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:20.850582 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:20.903301 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:20.967559 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:20.975355 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:21.035692 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 14:02:21.036366 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:21.061709 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240502 14:02:21.062368 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:21.075698 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:21.098903 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:21.131359 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:21.176674 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:21.210884 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:21.242249 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:21.364267 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:21.371743 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:21.495379 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:21.498364 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:21.535255 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:21.567615 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:21.572414 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:21.582341 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:21.640828 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:21.819682 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:21.858759 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:21.932690 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240502 14:02:21.971505 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:22.164728 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:22.175570 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:22.178794 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:22.226591 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:22.266682 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:22.283974 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:22.304204 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:22.417136 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:22.426756 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:22.556111 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:22.589530 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:22.670182 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:22.729981 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:02:22.764181 15418 maintenance_manager.cc:419] P d9f1f4817ca9494da13bfe388be84b9d: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033374
W20240502 14:02:22.765295 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:02:22.784554 15417 maintenance_manager.cc:643] P d9f1f4817ca9494da13bfe388be84b9d: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.020s user 0.007s sys 0.011s Metrics: {"bytes_written":20977,"cfile_init":1,"dirs.queue_time_us":194,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":1012,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":538,"lbm_writes_lt_1ms":29,"peak_mem_usage":0,"rows_written":43,"thread_start_us":158,"threads_started":2,"wal-append.queue_time_us":136}
I20240502 14:02:22.785503 15418 maintenance_manager.cc:419] P d9f1f4817ca9494da13bfe388be84b9d: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 11217 bytes on disk
I20240502 14:02:22.786067 15417 maintenance_manager.cc:643] P d9f1f4817ca9494da13bfe388be84b9d: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240502 14:02:22.901670 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:22.966864 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:23.027549 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:23.061187 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:23.194579 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:23.232908 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:23.253681 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:23.318311 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:23.322558 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:23.331840 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:23.343158 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:23.358729 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:23.446455 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:02:23.493873 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:23.527892 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:23.555039 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:23.587626 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:23.624698 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:23.680510 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:23.683167 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:23.825430 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:23.839357 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:23.849436 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:23.871186 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:23.905779 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:24.018330 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:24.049290 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:24.085491 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:24.103168 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:24.123178 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:24.260700 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:24.387790 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:24.421934 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:24.480124 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:24.504387 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:24.566504 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:24.567806 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:24.614524 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:24.849449 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:24.857755 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:24.889315 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:24.933400 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:24.961110 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:25.008651 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:25.009263 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:25.017275 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:25.170159 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:25.210343 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:25.351431 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:25.423480 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:25.428567 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:25.592468 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:25.640684 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:25.725112 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:25.739748 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240502 14:02:25.772936 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:25.781685 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:25.808773 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:25.821223 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:02:25.883435 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:25.964792 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:25.998340 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:26.020968 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:26.021921 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:26.035452 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:26.039161 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:26.073057 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 14:02:26.095134 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240502 14:02:26.109727 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:26.268975 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:26.278537 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:26.289093 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:26.310223 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:26.336952 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:26.427160 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:26.470209 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:26.475126 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:26.494706 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:26.496402 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:26.542732 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:26.603324 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:26.631192 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:26.643148 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:26.672421 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:26.862495 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:26.900210 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:27.105677 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:02:27.111126 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:27.122771 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:27.146574 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:27.244676 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:27.389986 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:27.436991 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:27.491194 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:27.609474 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:27.633869 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:27.648665 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:27.681593 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:27.701745 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:27.759171 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:27.806869 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:27.820370 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:02:27.944603 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling FlushMRSOp(d81f14b7c1d1468196d0fe6d67fe93a7): perf score=0.033338
I20240502 14:02:27.969712 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: FlushMRSOp(d81f14b7c1d1468196d0fe6d67fe93a7) complete. Timing: real 0.025s user 0.023s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":264,"dirs.queue_time_us":214,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1111,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":399,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":266,"threads_started":3,"wal-append.queue_time_us":150}
I20240502 14:02:27.971005 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling UndoDeltaBlockGCOp(d81f14b7c1d1468196d0fe6d67fe93a7): 4586 bytes on disk
I20240502 14:02:27.971530 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: UndoDeltaBlockGCOp(d81f14b7c1d1468196d0fe6d67fe93a7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240502 14:02:27.972321 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling FlushMRSOp(b07f4db6f82e4a56accdbf1bbbc5f965): perf score=0.033345
I20240502 14:02:27.986541 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling FlushMRSOp(9391f16ae2d247ec8a71e5c971cdda71): perf score=0.033353
I20240502 14:02:27.990790 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: FlushMRSOp(b07f4db6f82e4a56accdbf1bbbc5f965) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1598,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":398,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":105,"threads_started":1,"wal-append.queue_time_us":178}
I20240502 14:02:27.991868 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling UndoDeltaBlockGCOp(b07f4db6f82e4a56accdbf1bbbc5f965): 4840 bytes on disk
I20240502 14:02:27.992719 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: UndoDeltaBlockGCOp(b07f4db6f82e4a56accdbf1bbbc5f965) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240502 14:02:28.023594 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: FlushMRSOp(9391f16ae2d247ec8a71e5c971cdda71) complete. Timing: real 0.037s user 0.028s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":172,"dirs.queue_time_us":930,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":940,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":654,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":277,"threads_started":3,"wal-append.queue_time_us":179}
I20240502 14:02:28.024617 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling UndoDeltaBlockGCOp(9391f16ae2d247ec8a71e5c971cdda71): 4925 bytes on disk
I20240502 14:02:28.025228 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: UndoDeltaBlockGCOp(9391f16ae2d247ec8a71e5c971cdda71) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240502 14:02:28.027742 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling FlushMRSOp(9391f16ae2d247ec8a71e5c971cdda71): perf score=0.033361
I20240502 14:02:28.072818 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: FlushMRSOp(9391f16ae2d247ec8a71e5c971cdda71) complete. Timing: real 0.045s user 0.032s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":204,"dirs.queue_time_us":7280,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1370,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3701,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":343,"threads_started":3,"wal-append.queue_time_us":6552}
I20240502 14:02:28.074362 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling UndoDeltaBlockGCOp(9391f16ae2d247ec8a71e5c971cdda71): 4925 bytes on disk
I20240502 14:02:28.075039 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: UndoDeltaBlockGCOp(9391f16ae2d247ec8a71e5c971cdda71) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240502 14:02:28.089689 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling FlushMRSOp(b07f4db6f82e4a56accdbf1bbbc5f965): perf score=0.033383
W20240502 14:02:28.152209 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:02:28.152716 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: FlushMRSOp(b07f4db6f82e4a56accdbf1bbbc5f965) complete. Timing: real 0.063s user 0.028s sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":19291,"dirs.queue_time_us":14593,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":3510,"drs_written":1,"lbm_read_time_us":144,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4976,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":262,"threads_started":3,"wal-append.queue_time_us":4808}
I20240502 14:02:28.154013 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling UndoDeltaBlockGCOp(b07f4db6f82e4a56accdbf1bbbc5f965): 4840 bytes on disk
I20240502 14:02:28.154716 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: UndoDeltaBlockGCOp(b07f4db6f82e4a56accdbf1bbbc5f965) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240502 14:02:28.156793 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:28.175716 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:28.179119 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:28.201882 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:28.207978 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:28.246269 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:28.290706 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:28.346135 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:28.347594 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:02:28.398206 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:28.511322 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:28.559145 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:28.562491 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:28.585232 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:28.585788 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:28.590759 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:28.670436 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:28.705677 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:02:28.745733 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling FlushMRSOp(619a8ec66ba74ecea689c17b4734253a): perf score=0.033363
I20240502 14:02:28.756963 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: FlushMRSOp(619a8ec66ba74ecea689c17b4734253a) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":185,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1303,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":426,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":139,"threads_started":2,"wal-append.queue_time_us":140}
I20240502 14:02:28.757699 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling UndoDeltaBlockGCOp(619a8ec66ba74ecea689c17b4734253a): 4586 bytes on disk
I20240502 14:02:28.758013 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: UndoDeltaBlockGCOp(619a8ec66ba74ecea689c17b4734253a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240502 14:02:28.777959 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling FlushMRSOp(619a8ec66ba74ecea689c17b4734253a): perf score=0.033373
W20240502 14:02:28.778051 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:02:28.789597 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: FlushMRSOp(619a8ec66ba74ecea689c17b4734253a) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":168,"dirs.run_cpu_time_us":138,"dirs.run_wall_time_us":1428,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":441,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":137,"threads_started":2,"wal-append.queue_time_us":114}
I20240502 14:02:28.790478 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling FlushMRSOp(387bdd8ae3b34f588e2170b061a5bc05): perf score=0.033376
I20240502 14:02:28.801476 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: FlushMRSOp(387bdd8ae3b34f588e2170b061a5bc05) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":1203,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":360,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":75,"threads_started":1,"wal-append.queue_time_us":200}
I20240502 14:02:28.802201 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling UndoDeltaBlockGCOp(387bdd8ae3b34f588e2170b061a5bc05): 4925 bytes on disk
I20240502 14:02:28.802553 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: UndoDeltaBlockGCOp(387bdd8ae3b34f588e2170b061a5bc05) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
I20240502 14:02:28.803335 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling UndoDeltaBlockGCOp(619a8ec66ba74ecea689c17b4734253a): 4586 bytes on disk
I20240502 14:02:28.803725 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: UndoDeltaBlockGCOp(619a8ec66ba74ecea689c17b4734253a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240502 14:02:28.828044 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling FlushMRSOp(02d3da05d82340178e27218c2ce56b67): perf score=0.033388
I20240502 14:02:28.839612 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: FlushMRSOp(02d3da05d82340178e27218c2ce56b67) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":270,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1437,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":434,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":156,"threads_started":2,"wal-append.queue_time_us":145}
I20240502 14:02:28.840375 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling UndoDeltaBlockGCOp(02d3da05d82340178e27218c2ce56b67): 4838 bytes on disk
I20240502 14:02:28.840716 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: UndoDeltaBlockGCOp(02d3da05d82340178e27218c2ce56b67) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240502 14:02:28.872344 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:28.878715 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:28.924121 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:28.941695 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:28.950637 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:29.023350 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:02:29.091979 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling FlushMRSOp(48fbc670ec504d26a11bf1b9fbd78981): perf score=0.033336
I20240502 14:02:29.104458 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: FlushMRSOp(48fbc670ec504d26a11bf1b9fbd78981) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":119,"dirs.run_cpu_time_us":122,"dirs.run_wall_time_us":1427,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":452,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":71,"threads_started":1,"wal-append.queue_time_us":149}
I20240502 14:02:29.105310 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling UndoDeltaBlockGCOp(48fbc670ec504d26a11bf1b9fbd78981): 4586 bytes on disk
I20240502 14:02:29.105696 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: UndoDeltaBlockGCOp(48fbc670ec504d26a11bf1b9fbd78981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240502 14:02:29.120182 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:29.128844 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:02:29.158097 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling FlushMRSOp(a7981c3f46d8483caae82fbf2296e762): perf score=0.033352
W20240502 14:02:29.160816 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:29.161402 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:02:29.170548 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: FlushMRSOp(a7981c3f46d8483caae82fbf2296e762) complete. Timing: real 0.012s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":182,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1619,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":414,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":127,"threads_started":2,"wal-append.queue_time_us":133}
I20240502 14:02:29.171257 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling FlushMRSOp(48fbc670ec504d26a11bf1b9fbd78981): perf score=0.033354
I20240502 14:02:29.182534 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: FlushMRSOp(48fbc670ec504d26a11bf1b9fbd78981) complete. Timing: real 0.011s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1662,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":447,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":72,"threads_started":1,"wal-append.queue_time_us":173}
I20240502 14:02:29.183305 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling UndoDeltaBlockGCOp(a7981c3f46d8483caae82fbf2296e762): 4840 bytes on disk
I20240502 14:02:29.183638 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: UndoDeltaBlockGCOp(a7981c3f46d8483caae82fbf2296e762) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240502 14:02:29.184221 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling UndoDeltaBlockGCOp(48fbc670ec504d26a11bf1b9fbd78981): 4586 bytes on disk
I20240502 14:02:29.184736 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: UndoDeltaBlockGCOp(48fbc670ec504d26a11bf1b9fbd78981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":201,"lbm_reads_lt_1ms":4}
W20240502 14:02:29.231169 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:02:29.259939 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling FlushMRSOp(1f562ad00ec34a8d9184671ed55d84d1): perf score=0.033382
I20240502 14:02:29.271522 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: FlushMRSOp(1f562ad00ec34a8d9184671ed55d84d1) complete. Timing: real 0.011s user 0.008s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":159,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1621,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":437,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":154,"threads_started":2,"wal-append.queue_time_us":136}
I20240502 14:02:29.272264 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling UndoDeltaBlockGCOp(1f562ad00ec34a8d9184671ed55d84d1): 4925 bytes on disk
I20240502 14:02:29.272578 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: UndoDeltaBlockGCOp(1f562ad00ec34a8d9184671ed55d84d1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
W20240502 14:02:29.458190 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:29.481320 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:02:29.556336 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling FlushMRSOp(b6d365c10c76416a944ec8fe409e3dc5): perf score=0.033334
I20240502 14:02:29.567667 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: FlushMRSOp(b6d365c10c76416a944ec8fe409e3dc5) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":196,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1537,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":418,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":166,"threads_started":2,"wal-append.queue_time_us":129}
I20240502 14:02:29.568523 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling UndoDeltaBlockGCOp(b6d365c10c76416a944ec8fe409e3dc5): 4586 bytes on disk
I20240502 14:02:29.568893 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: UndoDeltaBlockGCOp(b6d365c10c76416a944ec8fe409e3dc5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240502 14:02:29.607398 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling FlushMRSOp(28e371c1a999431499e09479e8f5b44b): perf score=0.033347
I20240502 14:02:29.619133 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: FlushMRSOp(28e371c1a999431499e09479e8f5b44b) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":164,"dirs.run_cpu_time_us":150,"dirs.run_wall_time_us":1504,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":398,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":154,"threads_started":2,"wal-append.queue_time_us":132}
I20240502 14:02:29.619915 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling UndoDeltaBlockGCOp(28e371c1a999431499e09479e8f5b44b): 4840 bytes on disk
I20240502 14:02:29.620249 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: UndoDeltaBlockGCOp(28e371c1a999431499e09479e8f5b44b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240502 14:02:29.677160 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
I20240502 14:02:29.686983 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling FlushMRSOp(1f80c8590e994a10a9b7c6c174e28173): perf score=0.033370
W20240502 14:02:29.689651 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:02:29.699378 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: FlushMRSOp(1f80c8590e994a10a9b7c6c174e28173) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":159,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1328,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":440,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":181,"threads_started":2,"wal-append.queue_time_us":168}
I20240502 14:02:29.700119 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling UndoDeltaBlockGCOp(1f80c8590e994a10a9b7c6c174e28173): 4925 bytes on disk
I20240502 14:02:29.700471 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: UndoDeltaBlockGCOp(1f80c8590e994a10a9b7c6c174e28173) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240502 14:02:29.701074 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling FlushMRSOp(b6d365c10c76416a944ec8fe409e3dc5): perf score=0.033372
I20240502 14:02:29.714206 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: FlushMRSOp(b6d365c10c76416a944ec8fe409e3dc5) complete. Timing: real 0.013s user 0.010s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":1228,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":445,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":116,"threads_started":1,"wal-append.queue_time_us":203}
I20240502 14:02:29.714995 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling UndoDeltaBlockGCOp(b6d365c10c76416a944ec8fe409e3dc5): 4586 bytes on disk
I20240502 14:02:29.715399 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: UndoDeltaBlockGCOp(b6d365c10c76416a944ec8fe409e3dc5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 14:02:29.774520 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling FlushMRSOp(1f80c8590e994a10a9b7c6c174e28173): perf score=0.033396
I20240502 14:02:29.785907 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: FlushMRSOp(1f80c8590e994a10a9b7c6c174e28173) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":151,"dirs.run_cpu_time_us":122,"dirs.run_wall_time_us":1391,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":405,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":129,"threads_started":2,"wal-append.queue_time_us":138}
I20240502 14:02:29.786679 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling UndoDeltaBlockGCOp(1f80c8590e994a10a9b7c6c174e28173): 4925 bytes on disk
I20240502 14:02:29.787055 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: UndoDeltaBlockGCOp(1f80c8590e994a10a9b7c6c174e28173) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 14:02:29.787655 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling FlushMRSOp(28e371c1a999431499e09479e8f5b44b): perf score=0.033397
I20240502 14:02:29.800761 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: FlushMRSOp(28e371c1a999431499e09479e8f5b44b) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1281,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":495,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":67,"threads_started":1,"wal-append.queue_time_us":140}
I20240502 14:02:29.801602 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling UndoDeltaBlockGCOp(28e371c1a999431499e09479e8f5b44b): 4840 bytes on disk
I20240502 14:02:29.802032 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: UndoDeltaBlockGCOp(28e371c1a999431499e09479e8f5b44b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
W20240502 14:02:29.854489 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:29.920265 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:29.920673 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:30.001120 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:30.035692 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:30.046206 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:30.146639 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:30.155252 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:30.246192 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:02:30.303962 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling FlushMRSOp(bb9f100600a246708457280d91826fce): perf score=0.033338
I20240502 14:02:30.316133 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: FlushMRSOp(bb9f100600a246708457280d91826fce) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":157,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":1319,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":428,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":143,"threads_started":2,"wal-append.queue_time_us":128}
I20240502 14:02:30.316998 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling UndoDeltaBlockGCOp(bb9f100600a246708457280d91826fce): 4925 bytes on disk
I20240502 14:02:30.317339 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: UndoDeltaBlockGCOp(bb9f100600a246708457280d91826fce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240502 14:02:30.321674 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling FlushMRSOp(6e76bd0ecd3e4c168e141fb7984af3ec): perf score=0.033343
I20240502 14:02:30.332870 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: FlushMRSOp(6e76bd0ecd3e4c168e141fb7984af3ec) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":150,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1252,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":382,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":134,"threads_started":2,"wal-append.queue_time_us":133}
I20240502 14:02:30.333600 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling UndoDeltaBlockGCOp(6e76bd0ecd3e4c168e141fb7984af3ec): 4586 bytes on disk
I20240502 14:02:30.333940 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: UndoDeltaBlockGCOp(6e76bd0ecd3e4c168e141fb7984af3ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240502 14:02:30.334650 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling FlushMRSOp(2d92e5885bf94c3e8e0e4dee40017699): perf score=0.033345
W20240502 14:02:30.346455 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:02:30.347036 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: FlushMRSOp(2d92e5885bf94c3e8e0e4dee40017699) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":139,"dirs.run_wall_time_us":1283,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":361,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":83,"threads_started":1,"wal-append.queue_time_us":154}
I20240502 14:02:30.347754 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling UndoDeltaBlockGCOp(2d92e5885bf94c3e8e0e4dee40017699): 4840 bytes on disk
I20240502 14:02:30.348071 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: UndoDeltaBlockGCOp(2d92e5885bf94c3e8e0e4dee40017699) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240502 14:02:30.366047 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:02:30.372684 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling FlushMRSOp(bb9f100600a246708457280d91826fce): perf score=0.033357
I20240502 14:02:30.385505 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: FlushMRSOp(bb9f100600a246708457280d91826fce) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":196,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1517,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":452,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":896,"thread_start_us":169,"threads_started":2,"wal-append.queue_time_us":146}
I20240502 14:02:30.386260 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling FlushMRSOp(2d92e5885bf94c3e8e0e4dee40017699): perf score=0.033359
I20240502 14:02:30.398504 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: FlushMRSOp(2d92e5885bf94c3e8e0e4dee40017699) complete. Timing: real 0.012s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1422,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":431,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":80,"threads_started":1,"wal-append.queue_time_us":165}
I20240502 14:02:30.399279 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling UndoDeltaBlockGCOp(bb9f100600a246708457280d91826fce): 4925 bytes on disk
I20240502 14:02:30.399610 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: UndoDeltaBlockGCOp(bb9f100600a246708457280d91826fce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240502 14:02:30.400317 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling UndoDeltaBlockGCOp(2d92e5885bf94c3e8e0e4dee40017699): 4840 bytes on disk
I20240502 14:02:30.400645 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: UndoDeltaBlockGCOp(2d92e5885bf94c3e8e0e4dee40017699) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
W20240502 14:02:30.431931 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:02:30.468232 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling FlushMRSOp(6e76bd0ecd3e4c168e141fb7984af3ec): perf score=0.033379
I20240502 14:02:30.479442 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: FlushMRSOp(6e76bd0ecd3e4c168e141fb7984af3ec) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":152,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1632,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":392,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":146,"threads_started":2,"wal-append.queue_time_us":139}
I20240502 14:02:30.480166 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling UndoDeltaBlockGCOp(6e76bd0ecd3e4c168e141fb7984af3ec): 4586 bytes on disk
I20240502 14:02:30.480508 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: UndoDeltaBlockGCOp(6e76bd0ecd3e4c168e141fb7984af3ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240502 14:02:30.526786 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:30.664507 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:30.667116 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:30.674456 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:30.687772 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:30.720129 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:30.725353 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:30.774497 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 125 similar messages]
W20240502 14:02:30.780798 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:02:30.810365 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:02:30.819283 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling FlushMRSOp(436c127af87c49e8b7bf9b6571f7dc30): perf score=0.033343
I20240502 14:02:30.831121 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: FlushMRSOp(436c127af87c49e8b7bf9b6571f7dc30) complete. Timing: real 0.012s user 0.006s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":122,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1528,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":456,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":121,"threads_started":2,"wal-append.queue_time_us":130}
I20240502 14:02:30.831851 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling UndoDeltaBlockGCOp(436c127af87c49e8b7bf9b6571f7dc30): 4840 bytes on disk
I20240502 14:02:30.832154 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: UndoDeltaBlockGCOp(436c127af87c49e8b7bf9b6571f7dc30) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240502 14:02:30.832749 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling FlushMRSOp(b0cd21dc897b41aca9118be6df48cc0c): perf score=0.033346
I20240502 14:02:30.842988 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: FlushMRSOp(b0cd21dc897b41aca9118be6df48cc0c) complete. Timing: real 0.010s user 0.009s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1280,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":389,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":79,"threads_started":1,"wal-append.queue_time_us":152}
I20240502 14:02:30.843713 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling UndoDeltaBlockGCOp(b0cd21dc897b41aca9118be6df48cc0c): 4586 bytes on disk
I20240502 14:02:30.844007 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: UndoDeltaBlockGCOp(b0cd21dc897b41aca9118be6df48cc0c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240502 14:02:30.850045 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling FlushMRSOp(2886dfbec1e141198b0cf7a435fac498): perf score=0.033355
I20240502 14:02:30.862466 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: FlushMRSOp(2886dfbec1e141198b0cf7a435fac498) complete. Timing: real 0.012s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":155,"dirs.run_cpu_time_us":136,"dirs.run_wall_time_us":1284,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":376,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":131,"threads_started":2,"wal-append.queue_time_us":123}
I20240502 14:02:30.863340 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling FlushMRSOp(b0cd21dc897b41aca9118be6df48cc0c): perf score=0.033356
I20240502 14:02:30.873678 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: FlushMRSOp(b0cd21dc897b41aca9118be6df48cc0c) complete. Timing: real 0.010s user 0.009s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":144,"dirs.run_wall_time_us":1261,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":356,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":118,"threads_started":1,"wal-append.queue_time_us":193}
I20240502 14:02:30.874466 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling UndoDeltaBlockGCOp(2886dfbec1e141198b0cf7a435fac498): 4924 bytes on disk
I20240502 14:02:30.874788 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: UndoDeltaBlockGCOp(2886dfbec1e141198b0cf7a435fac498) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240502 14:02:30.875358 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling UndoDeltaBlockGCOp(b0cd21dc897b41aca9118be6df48cc0c): 4586 bytes on disk
I20240502 14:02:30.875633 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: UndoDeltaBlockGCOp(b0cd21dc897b41aca9118be6df48cc0c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240502 14:02:30.895630 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:30.897817 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:02:30.902426 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling FlushMRSOp(2886dfbec1e141198b0cf7a435fac498): perf score=0.033368
I20240502 14:02:30.914050 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: FlushMRSOp(2886dfbec1e141198b0cf7a435fac498) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":148,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1508,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":387,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":135,"threads_started":2,"wal-append.queue_time_us":137}
I20240502 14:02:30.914803 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling UndoDeltaBlockGCOp(2886dfbec1e141198b0cf7a435fac498): 4924 bytes on disk
I20240502 14:02:30.915120 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: UndoDeltaBlockGCOp(2886dfbec1e141198b0cf7a435fac498) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240502 14:02:30.932363 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:30.935173 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:30.937081 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:30.953028 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:02:30.982349 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling FlushMRSOp(436c127af87c49e8b7bf9b6571f7dc30): perf score=0.033387
I20240502 14:02:30.993667 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: FlushMRSOp(436c127af87c49e8b7bf9b6571f7dc30) complete. Timing: real 0.011s user 0.008s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":161,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1480,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":413,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":132,"threads_started":2,"wal-append.queue_time_us":122}
I20240502 14:02:30.994525 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling UndoDeltaBlockGCOp(436c127af87c49e8b7bf9b6571f7dc30): 4840 bytes on disk
I20240502 14:02:30.994860 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: UndoDeltaBlockGCOp(436c127af87c49e8b7bf9b6571f7dc30) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240502 14:02:31.098610 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:31.127740 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:02:31.128350 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:31.135725 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:31.154361 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:31.199258 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:31.209554 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:31.309239 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:31.319574 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:02:31.335321 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:31.335508 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:02:31.345950 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling FlushMRSOp(dfdcf1124e8a474facc14c45e8963568): perf score=0.033382
I20240502 14:02:31.357910 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: FlushMRSOp(dfdcf1124e8a474facc14c45e8963568) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":160,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":1718,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":471,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":125,"threads_started":2,"wal-append.queue_time_us":122}
I20240502 14:02:31.358716 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling FlushMRSOp(837e8b1235a44131bdaef24b763d1cac): perf score=0.033384
I20240502 14:02:31.370534 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: FlushMRSOp(837e8b1235a44131bdaef24b763d1cac) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":1340,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":431,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":113,"threads_started":1,"wal-append.queue_time_us":206}
I20240502 14:02:31.371392 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling UndoDeltaBlockGCOp(dfdcf1124e8a474facc14c45e8963568): 4925 bytes on disk
I20240502 14:02:31.371826 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: UndoDeltaBlockGCOp(dfdcf1124e8a474facc14c45e8963568) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240502 14:02:31.372402 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling UndoDeltaBlockGCOp(837e8b1235a44131bdaef24b763d1cac): 4840 bytes on disk
I20240502 14:02:31.372727 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: UndoDeltaBlockGCOp(837e8b1235a44131bdaef24b763d1cac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
I20240502 14:02:31.377444 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling FlushMRSOp(837e8b1235a44131bdaef24b763d1cac): perf score=0.033391
I20240502 14:02:31.390067 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: FlushMRSOp(837e8b1235a44131bdaef24b763d1cac) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":159,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1370,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":433,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":142}
I20240502 14:02:31.391206 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling UndoDeltaBlockGCOp(837e8b1235a44131bdaef24b763d1cac): 4840 bytes on disk
I20240502 14:02:31.391641 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: UndoDeltaBlockGCOp(837e8b1235a44131bdaef24b763d1cac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
W20240502 14:02:31.401978 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:02:31.416997 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling FlushMRSOp(22eef55211ca4f65979022d2f4ae829f): perf score=0.033400
W20240502 14:02:31.420368 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:02:31.428897 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: FlushMRSOp(22eef55211ca4f65979022d2f4ae829f) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":175,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1674,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":468,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":151,"threads_started":2,"wal-append.queue_time_us":140}
I20240502 14:02:31.429653 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling UndoDeltaBlockGCOp(22eef55211ca4f65979022d2f4ae829f): 4586 bytes on disk
I20240502 14:02:31.430007 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: UndoDeltaBlockGCOp(22eef55211ca4f65979022d2f4ae829f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240502 14:02:31.633446 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:31.647877 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
I20240502 14:02:31.681309 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling FlushMRSOp(1a5e33f337bd47b2a6b11b29a5eacad8): perf score=0.033343
W20240502 14:02:31.683338 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:02:31.693334 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: FlushMRSOp(1a5e33f337bd47b2a6b11b29a5eacad8) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":65,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1568,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":468,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":67,"threads_started":1,"wal-append.queue_time_us":178}
I20240502 14:02:31.694126 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling FlushMRSOp(03dd735113ab4ea6bdb696b11a4efd7f): perf score=0.033345
I20240502 14:02:31.705557 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: FlushMRSOp(03dd735113ab4ea6bdb696b11a4efd7f) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":72,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1257,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":394,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":79,"threads_started":1,"wal-append.queue_time_us":165}
I20240502 14:02:31.706418 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling UndoDeltaBlockGCOp(1a5e33f337bd47b2a6b11b29a5eacad8): 4840 bytes on disk
I20240502 14:02:31.706789 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: UndoDeltaBlockGCOp(1a5e33f337bd47b2a6b11b29a5eacad8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 14:02:31.707376 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling UndoDeltaBlockGCOp(03dd735113ab4ea6bdb696b11a4efd7f): 4586 bytes on disk
I20240502 14:02:31.707664 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: UndoDeltaBlockGCOp(03dd735113ab4ea6bdb696b11a4efd7f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240502 14:02:31.713893 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:02:31.747318 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling FlushMRSOp(76f07464d9bd49a9b1b9d58f6a180cf8): perf score=0.033361
I20240502 14:02:31.759282 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: FlushMRSOp(76f07464d9bd49a9b1b9d58f6a180cf8) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":172,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":1490,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":469,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":138}
I20240502 14:02:31.760172 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling UndoDeltaBlockGCOp(76f07464d9bd49a9b1b9d58f6a180cf8): 4925 bytes on disk
I20240502 14:02:31.760576 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: UndoDeltaBlockGCOp(76f07464d9bd49a9b1b9d58f6a180cf8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
W20240502 14:02:31.790688 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:31.839324 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:02:31.874709 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling FlushMRSOp(76f07464d9bd49a9b1b9d58f6a180cf8): perf score=0.033398
W20240502 14:02:31.879853 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:02:31.886631 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: FlushMRSOp(76f07464d9bd49a9b1b9d58f6a180cf8) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":159,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":1531,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":407,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":139}
I20240502 14:02:31.887354 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling UndoDeltaBlockGCOp(76f07464d9bd49a9b1b9d58f6a180cf8): 4925 bytes on disk
I20240502 14:02:31.887714 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: UndoDeltaBlockGCOp(76f07464d9bd49a9b1b9d58f6a180cf8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240502 14:02:31.893581 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling FlushMRSOp(1a5e33f337bd47b2a6b11b29a5eacad8): perf score=0.033400
I20240502 14:02:31.905725 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: FlushMRSOp(1a5e33f337bd47b2a6b11b29a5eacad8) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":152,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":1323,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":399,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":127,"threads_started":2,"wal-append.queue_time_us":125}
I20240502 14:02:31.906507 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling UndoDeltaBlockGCOp(1a5e33f337bd47b2a6b11b29a5eacad8): 4840 bytes on disk
I20240502 14:02:31.906833 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: UndoDeltaBlockGCOp(1a5e33f337bd47b2a6b11b29a5eacad8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240502 14:02:32.057708 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:32.100679 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:32.160454 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 47 similar messages]
I20240502 14:02:32.209504 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling FlushMRSOp(186a946321e445b58b0097c658847ebe): perf score=0.033350
I20240502 14:02:32.224350 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: FlushMRSOp(186a946321e445b58b0097c658847ebe) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":178,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1539,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":458,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":174,"threads_started":2,"wal-append.queue_time_us":135}
I20240502 14:02:32.225245 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling UndoDeltaBlockGCOp(186a946321e445b58b0097c658847ebe): 4925 bytes on disk
I20240502 14:02:32.225589 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: UndoDeltaBlockGCOp(186a946321e445b58b0097c658847ebe) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240502 14:02:32.226327 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling FlushMRSOp(7c57cab68b764db0917ed98e74757ff0): perf score=0.033350
I20240502 14:02:32.237637 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: FlushMRSOp(7c57cab68b764db0917ed98e74757ff0) complete. Timing: real 0.011s user 0.005s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1251,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":390,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":94,"threads_started":1,"wal-append.queue_time_us":166}
I20240502 14:02:32.238425 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling UndoDeltaBlockGCOp(7c57cab68b764db0917ed98e74757ff0): 4840 bytes on disk
I20240502 14:02:32.238718 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: UndoDeltaBlockGCOp(7c57cab68b764db0917ed98e74757ff0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240502 14:02:32.262413 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling FlushMRSOp(e51db70f5f284c4da3c1669fddb41b00): perf score=0.033361
W20240502 14:02:32.268649 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
I20240502 14:02:32.275017 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: FlushMRSOp(e51db70f5f284c4da3c1669fddb41b00) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":170,"dirs.run_cpu_time_us":148,"dirs.run_wall_time_us":1250,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":559,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":137,"threads_started":2,"wal-append.queue_time_us":148}
I20240502 14:02:32.275866 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling UndoDeltaBlockGCOp(e51db70f5f284c4da3c1669fddb41b00): 4586 bytes on disk
I20240502 14:02:32.276643 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: UndoDeltaBlockGCOp(e51db70f5f284c4da3c1669fddb41b00) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20240502 14:02:32.341415 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:32.356025 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:32.385944 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:02:32.389725 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling FlushMRSOp(186a946321e445b58b0097c658847ebe): perf score=0.033397
I20240502 14:02:32.401686 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: FlushMRSOp(186a946321e445b58b0097c658847ebe) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":158,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1482,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":473,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":133,"threads_started":2,"wal-append.queue_time_us":127}
I20240502 14:02:32.402489 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling UndoDeltaBlockGCOp(186a946321e445b58b0097c658847ebe): 4925 bytes on disk
I20240502 14:02:32.402838 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: UndoDeltaBlockGCOp(186a946321e445b58b0097c658847ebe) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 14:02:32.408643 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling FlushMRSOp(7c57cab68b764db0917ed98e74757ff0): perf score=0.033402
I20240502 14:02:32.419759 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: FlushMRSOp(7c57cab68b764db0917ed98e74757ff0) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":194,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1506,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":398,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":152,"threads_started":2,"wal-append.queue_time_us":189}
I20240502 14:02:32.420500 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling UndoDeltaBlockGCOp(7c57cab68b764db0917ed98e74757ff0): 4840 bytes on disk
I20240502 14:02:32.420826 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: UndoDeltaBlockGCOp(7c57cab68b764db0917ed98e74757ff0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240502 14:02:32.550980 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:32.567304 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:32.602814 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:32.637925 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:32.653306 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:32.682642 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
I20240502 14:02:32.778584 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling FlushMRSOp(9870bc797f184a6284ff6ca4f78b3826): perf score=0.033342
I20240502 14:02:32.789669 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: FlushMRSOp(9870bc797f184a6284ff6ca4f78b3826) complete. Timing: real 0.011s user 0.009s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":181,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1400,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":459,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":131,"threads_started":2,"wal-append.queue_time_us":117}
I20240502 14:02:32.790419 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling UndoDeltaBlockGCOp(9870bc797f184a6284ff6ca4f78b3826): 4840 bytes on disk
I20240502 14:02:32.790745 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: UndoDeltaBlockGCOp(9870bc797f184a6284ff6ca4f78b3826) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240502 14:02:32.865725 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:02:32.904810 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling FlushMRSOp(9870bc797f184a6284ff6ca4f78b3826): perf score=0.033376
I20240502 14:02:32.916549 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: FlushMRSOp(9870bc797f184a6284ff6ca4f78b3826) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":177,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":1596,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":427,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":146,"threads_started":2,"wal-append.queue_time_us":144}
I20240502 14:02:32.917371 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling UndoDeltaBlockGCOp(9870bc797f184a6284ff6ca4f78b3826): 4840 bytes on disk
I20240502 14:02:32.917707 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: UndoDeltaBlockGCOp(9870bc797f184a6284ff6ca4f78b3826) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240502 14:02:32.922709 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling FlushMRSOp(e48dda3e62e14e46b03d79648e3e2773): perf score=0.033380
W20240502 14:02:32.934163 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:02:32.934104 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: FlushMRSOp(e48dda3e62e14e46b03d79648e3e2773) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":173,"dirs.run_cpu_time_us":265,"dirs.run_wall_time_us":1409,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":394,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":129,"threads_started":2,"wal-append.queue_time_us":131}
I20240502 14:02:32.934859 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling UndoDeltaBlockGCOp(e48dda3e62e14e46b03d79648e3e2773): 4586 bytes on disk
I20240502 14:02:32.935189 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: UndoDeltaBlockGCOp(e48dda3e62e14e46b03d79648e3e2773) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
W20240502 14:02:32.939833 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:02:32.991314 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling FlushMRSOp(1c004727e7c64f33bc9492184b77fde9): perf score=0.033402
I20240502 14:02:33.004096 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: FlushMRSOp(1c004727e7c64f33bc9492184b77fde9) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":175,"dirs.run_cpu_time_us":144,"dirs.run_wall_time_us":1531,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":419,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":144,"threads_started":2,"wal-append.queue_time_us":131}
I20240502 14:02:33.004848 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling UndoDeltaBlockGCOp(1c004727e7c64f33bc9492184b77fde9): 4925 bytes on disk
I20240502 14:02:33.005208 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: UndoDeltaBlockGCOp(1c004727e7c64f33bc9492184b77fde9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
W20240502 14:02:33.081676 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:33.094519 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:33.100219 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:33.160687 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:33.214380 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:33.225033 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:02:33.238505 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:33.240103 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:33.248893 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:33.256747 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:33.259585 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:33.317610 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:33.344235 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:33.422901 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:33.439229 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:33.460487 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:33.562060 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:33.643344 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:33.712743 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:33.745088 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:33.777128 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:33.777217 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:33.785104 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:33.835132 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:33.900853 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:33.981683 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:34.197127 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:34.246281 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:34.260916 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:34.266037 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:34.269160 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:34.420487 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:34.444881 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:34.643388 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:34.693248 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:34.740134 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:34.846933 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:34.861629 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:34.918452 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:35.040798 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:35.048102 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:35.055665 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:35.169304 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:35.192693 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:35.220252 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:35.379180 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:35.379909 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:35.470484 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:35.593417 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:35.625124 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:35.626021 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:35.628221 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:35.655866 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:35.678992 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:35.687994 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:02:35.782786 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:02:35.783933 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:35.785156 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:35.793155 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:35.835929 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:35.928803 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:35.954823 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:35.971204 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:35.978219 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:36.054306 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:36.095546 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:36.098862 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:36.116811 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:36.145262 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:02:36.168834 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:36.179246 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:36.215493 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:36.353863 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 14:02:36.354475 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:36.466189 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:36.474488 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:36.535907 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:36.560680 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:02:36.668774 23361 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:02:36.668906 23361 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:02:36.669397 23361 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:02:36.670316 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:36.670451 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:02:36.670512 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:02:36.670641 23361 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:02:36.726488 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:36.746863 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:02:36.778347 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling FlushMRSOp(1a5e33f337bd47b2a6b11b29a5eacad8): perf score=0.033376
I20240502 14:02:36.802799 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: FlushMRSOp(1a5e33f337bd47b2a6b11b29a5eacad8) complete. Timing: real 0.024s user 0.024s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":185,"dirs.queue_time_us":297,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":988,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":398,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":285,"threads_started":3,"wal-append.queue_time_us":172}
I20240502 14:02:36.804203 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling FlushMRSOp(619a8ec66ba74ecea689c17b4734253a): perf score=0.033359
I20240502 14:02:36.818790 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: FlushMRSOp(619a8ec66ba74ecea689c17b4734253a) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1440,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":391,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":95,"threads_started":1,"wal-append.queue_time_us":178}
I20240502 14:02:36.819576 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling FlushMRSOp(d81f14b7c1d1468196d0fe6d67fe93a7): perf score=0.033342
I20240502 14:02:36.832072 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: FlushMRSOp(d81f14b7c1d1468196d0fe6d67fe93a7) complete. Timing: real 0.012s user 0.008s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1390,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":358,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":113,"threads_started":1,"wal-append.queue_time_us":205}
I20240502 14:02:36.833248 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling UndoDeltaBlockGCOp(1a5e33f337bd47b2a6b11b29a5eacad8): 4840 bytes on disk
I20240502 14:02:36.833889 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: UndoDeltaBlockGCOp(1a5e33f337bd47b2a6b11b29a5eacad8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240502 14:02:36.834666 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling UndoDeltaBlockGCOp(619a8ec66ba74ecea689c17b4734253a): 4586 bytes on disk
I20240502 14:02:36.834973 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: UndoDeltaBlockGCOp(619a8ec66ba74ecea689c17b4734253a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240502 14:02:36.835554 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling UndoDeltaBlockGCOp(d81f14b7c1d1468196d0fe6d67fe93a7): 4586 bytes on disk
I20240502 14:02:36.835826 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: UndoDeltaBlockGCOp(d81f14b7c1d1468196d0fe6d67fe93a7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4}
W20240502 14:02:36.851668 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:36.974735 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:36.991300 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:37.016554 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:37.029850 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:37.054368 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:37.074018 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:02:37.086958 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling FlushMRSOp(2d92e5885bf94c3e8e0e4dee40017699): perf score=0.033339
I20240502 14:02:37.099344 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: FlushMRSOp(2d92e5885bf94c3e8e0e4dee40017699) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1573,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":431,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":67,"threads_started":1,"wal-append.queue_time_us":151}
I20240502 14:02:37.100309 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling FlushMRSOp(03dd735113ab4ea6bdb696b11a4efd7f): perf score=0.033341
I20240502 14:02:37.110899 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: FlushMRSOp(03dd735113ab4ea6bdb696b11a4efd7f) complete. Timing: real 0.010s user 0.007s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":1249,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":352,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":101,"threads_started":1,"wal-append.queue_time_us":177}
I20240502 14:02:37.111788 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling UndoDeltaBlockGCOp(2d92e5885bf94c3e8e0e4dee40017699): 4840 bytes on disk
I20240502 14:02:37.112167 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: UndoDeltaBlockGCOp(2d92e5885bf94c3e8e0e4dee40017699) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240502 14:02:37.112723 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling UndoDeltaBlockGCOp(03dd735113ab4ea6bdb696b11a4efd7f): 4586 bytes on disk
I20240502 14:02:37.113005 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: UndoDeltaBlockGCOp(03dd735113ab4ea6bdb696b11a4efd7f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
I20240502 14:02:37.364220 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling FlushMRSOp(a7981c3f46d8483caae82fbf2296e762): perf score=0.033389
I20240502 14:02:37.376263 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: FlushMRSOp(a7981c3f46d8483caae82fbf2296e762) complete. Timing: real 0.012s user 0.009s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":73,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":1607,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":403,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":120,"threads_started":1,"wal-append.queue_time_us":220}
I20240502 14:02:37.377000 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling FlushMRSOp(7c57cab68b764db0917ed98e74757ff0): perf score=0.033344
I20240502 14:02:37.390708 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: FlushMRSOp(7c57cab68b764db0917ed98e74757ff0) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":71,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":1463,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":380,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":101,"threads_started":1,"wal-append.queue_time_us":184}
I20240502 14:02:37.391865 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling FlushMRSOp(22eef55211ca4f65979022d2f4ae829f): perf score=0.033346
I20240502 14:02:37.403414 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: FlushMRSOp(22eef55211ca4f65979022d2f4ae829f) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1345,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":399,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":111,"threads_started":1,"wal-append.queue_time_us":186}
I20240502 14:02:37.404219 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling UndoDeltaBlockGCOp(7c57cab68b764db0917ed98e74757ff0): 4840 bytes on disk
I20240502 14:02:37.404588 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: UndoDeltaBlockGCOp(7c57cab68b764db0917ed98e74757ff0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240502 14:02:37.405359 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling UndoDeltaBlockGCOp(a7981c3f46d8483caae82fbf2296e762): 4840 bytes on disk
I20240502 14:02:37.405737 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: UndoDeltaBlockGCOp(a7981c3f46d8483caae82fbf2296e762) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 14:02:37.406445 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling UndoDeltaBlockGCOp(22eef55211ca4f65979022d2f4ae829f): 4586 bytes on disk
I20240502 14:02:37.406756 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: UndoDeltaBlockGCOp(22eef55211ca4f65979022d2f4ae829f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4}
W20240502 14:02:37.417307 15898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 14:02:37.417917 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:37.439705 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:37.482261 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:37.536317 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:37.543156 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:37.546517 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:37.548218 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:37.663560 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:37.750375 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:37.793615 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:37.916801 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:37.988415 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:37.997702 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:38.029839 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:38.085760 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:38.093500 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:38.115512 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:38.128940 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:38.135315 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:38.179690 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:38.187147 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:38.222484 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:38.331961 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:38.360612 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:02:38.409734 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling FlushMRSOp(6e76bd0ecd3e4c168e141fb7984af3ec): perf score=0.033400
I20240502 14:02:38.421126 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: FlushMRSOp(6e76bd0ecd3e4c168e141fb7984af3ec) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":177,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":1701,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":400,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":146,"threads_started":2,"wal-append.queue_time_us":144}
I20240502 14:02:38.421876 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling UndoDeltaBlockGCOp(6e76bd0ecd3e4c168e141fb7984af3ec): 4586 bytes on disk
I20240502 14:02:38.422250 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: UndoDeltaBlockGCOp(6e76bd0ecd3e4c168e141fb7984af3ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240502 14:02:38.422919 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling FlushMRSOp(e48dda3e62e14e46b03d79648e3e2773): perf score=0.033401
I20240502 14:02:38.433574 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: FlushMRSOp(e48dda3e62e14e46b03d79648e3e2773) complete. Timing: real 0.010s user 0.009s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":1344,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":442,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":71,"threads_started":1,"wal-append.queue_time_us":135}
I20240502 14:02:38.434378 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling UndoDeltaBlockGCOp(e48dda3e62e14e46b03d79648e3e2773): 4586 bytes on disk
I20240502 14:02:38.434700 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: UndoDeltaBlockGCOp(e48dda3e62e14e46b03d79648e3e2773) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240502 14:02:38.435276 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling FlushMRSOp(28e371c1a999431499e09479e8f5b44b): perf score=0.033387
I20240502 14:02:38.445982 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: FlushMRSOp(28e371c1a999431499e09479e8f5b44b) complete. Timing: real 0.011s user 0.009s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1351,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":383,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":76,"threads_started":1,"wal-append.queue_time_us":168}
I20240502 14:02:38.446779 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling UndoDeltaBlockGCOp(28e371c1a999431499e09479e8f5b44b): 4840 bytes on disk
I20240502 14:02:38.447099 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: UndoDeltaBlockGCOp(28e371c1a999431499e09479e8f5b44b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
W20240502 14:02:38.449137 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:38.474267 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:38.491729 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:02:38.497953 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:38.523196 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:38.535301 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:38.545476 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:38.670388 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:38.674333 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:38.685652 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:38.716575 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:38.779542 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:38.859962 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:38.923894 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:39.059491 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:39.080549 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:39.301438 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:39.338008 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:39.398069 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:02:39.450246 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling FlushMRSOp(436c127af87c49e8b7bf9b6571f7dc30): perf score=0.033367
W20240502 14:02:39.455152 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:39.461421 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:02:39.465441 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: FlushMRSOp(436c127af87c49e8b7bf9b6571f7dc30) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":147,"dirs.run_cpu_time_us":123,"dirs.run_wall_time_us":1449,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":728,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":133,"threads_started":2,"wal-append.queue_time_us":129}
I20240502 14:02:39.466171 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling UndoDeltaBlockGCOp(436c127af87c49e8b7bf9b6571f7dc30): 4840 bytes on disk
I20240502 14:02:39.466547 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: UndoDeltaBlockGCOp(436c127af87c49e8b7bf9b6571f7dc30) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240502 14:02:39.570648 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:39.601727 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:02:39.717938 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling FlushMRSOp(2886dfbec1e141198b0cf7a435fac498): perf score=0.033335
W20240502 14:02:39.730201 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:02:39.731941 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: FlushMRSOp(2886dfbec1e141198b0cf7a435fac498) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1538,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":543,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":71,"threads_started":1,"wal-append.queue_time_us":151}
I20240502 14:02:39.732671 17562 maintenance_manager.cc:419] P 8fea283177394bd28d7eacf4fb744d59: Scheduling UndoDeltaBlockGCOp(2886dfbec1e141198b0cf7a435fac498): 4924 bytes on disk
I20240502 14:02:39.733007 17452 maintenance_manager.cc:643] P 8fea283177394bd28d7eacf4fb744d59: UndoDeltaBlockGCOp(2886dfbec1e141198b0cf7a435fac498) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
W20240502 14:02:39.787333 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:39.810822 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:02:39.810928 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling FlushMRSOp(22eef55211ca4f65979022d2f4ae829f): perf score=0.033380
W20240502 14:02:39.820027 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
I20240502 14:02:39.823323 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: FlushMRSOp(22eef55211ca4f65979022d2f4ae829f) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":198,"dirs.run_cpu_time_us":238,"dirs.run_wall_time_us":1738,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":458,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":163,"threads_started":2,"wal-append.queue_time_us":139}
I20240502 14:02:39.824059 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling UndoDeltaBlockGCOp(22eef55211ca4f65979022d2f4ae829f): 4586 bytes on disk
I20240502 14:02:39.824399 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: UndoDeltaBlockGCOp(22eef55211ca4f65979022d2f4ae829f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240502 14:02:39.904412 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:39.955281 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:40.008065 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:40.192365 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:40.203433 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:02:40.206174 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling FlushMRSOp(bb9f100600a246708457280d91826fce): perf score=0.033336
I20240502 14:02:40.219928 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: FlushMRSOp(bb9f100600a246708457280d91826fce) complete. Timing: real 0.014s user 0.008s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":187,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1706,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":510,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":145,"threads_started":2,"wal-append.queue_time_us":155}
I20240502 14:02:40.220944 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling UndoDeltaBlockGCOp(bb9f100600a246708457280d91826fce): 4925 bytes on disk
I20240502 14:02:40.221313 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: UndoDeltaBlockGCOp(bb9f100600a246708457280d91826fce) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
W20240502 14:02:40.259672 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:40.265962 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:40.314183 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
I20240502 14:02:40.326453 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling FlushMRSOp(dfdcf1124e8a474facc14c45e8963568): perf score=0.033370
I20240502 14:02:40.338269 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: FlushMRSOp(dfdcf1124e8a474facc14c45e8963568) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":192,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":1807,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":423,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":175,"threads_started":2,"wal-append.queue_time_us":177}
I20240502 14:02:40.339098 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling UndoDeltaBlockGCOp(dfdcf1124e8a474facc14c45e8963568): 4925 bytes on disk
I20240502 14:02:40.339466 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: UndoDeltaBlockGCOp(dfdcf1124e8a474facc14c45e8963568) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240502 14:02:40.363560 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:40.458109 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:40.471478 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:40.492211 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:40.514716 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:40.518018 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:02:40.590764 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling FlushMRSOp(02d3da05d82340178e27218c2ce56b67): perf score=0.033345
W20240502 14:02:40.593402 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:02:40.603006 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: FlushMRSOp(02d3da05d82340178e27218c2ce56b67) complete. Timing: real 0.012s user 0.009s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1613,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":426,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":102,"threads_started":1,"wal-append.queue_time_us":190}
I20240502 14:02:40.603765 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling UndoDeltaBlockGCOp(02d3da05d82340178e27218c2ce56b67): 4838 bytes on disk
I20240502 14:02:40.604167 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: UndoDeltaBlockGCOp(02d3da05d82340178e27218c2ce56b67) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20240502 14:02:40.605813 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:40.619566 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:40.665478 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:02:40.723393 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling FlushMRSOp(48fbc670ec504d26a11bf1b9fbd78981): perf score=0.033388
I20240502 14:02:40.736451 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: FlushMRSOp(48fbc670ec504d26a11bf1b9fbd78981) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":183,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1556,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":494,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":141,"threads_started":2,"wal-append.queue_time_us":131}
I20240502 14:02:40.737224 16011 maintenance_manager.cc:419] P 4d0f8f311e834d81bd54b9f8e7923bcd: Scheduling UndoDeltaBlockGCOp(48fbc670ec504d26a11bf1b9fbd78981): 4586 bytes on disk
I20240502 14:02:40.737619 15901 maintenance_manager.cc:643] P 4d0f8f311e834d81bd54b9f8e7923bcd: UndoDeltaBlockGCOp(48fbc670ec504d26a11bf1b9fbd78981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240502 14:02:40.741309 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:40.790235 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:40.809442 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:40.811647 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:02:40.847741 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:40.935371 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:40.936547 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:40.940100 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:02:41.041678 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:41.111789 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:41.132220 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:41.196775 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:41.209812 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:02:41.210476 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:41.246476 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:41.311848 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:41.320025 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:41.329700 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:41.384158 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:41.438454 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:02:41.541702 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:41.677734 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:41.768235 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:41.788807 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:41.832412 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:41.873313 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:42.021538 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:42.072517 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:42.094343 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:42.169926 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:42.252512 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:42.326031 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:42.358851 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:42.430011 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:42.447656 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:42.452077 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:42.571892 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 14:02:42.657999 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:42.742139 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:42.751780 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:42.804891 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:42.923797 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:42.947731 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:42.989293 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:43.007825 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:43.047626 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:43.102794 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:43.102996 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:43.105906 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:43.120440 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:43.182649 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:43.224057 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:43.266391 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:43.339756 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:43.353574 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:43.369019 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:43.425112 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:43.478197 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:43.502661 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:43.517112 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:02:43.535140 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:43.645423 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:43.687947 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:43.712154 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:43.821027 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:43.850900 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:43.854033 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:43.860883 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:43.902192 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:44.084311 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:44.147169 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:44.360934 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:44.385360 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:44.434115 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:44.550488 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:44.583914 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:44.662477 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:44.720770 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:44.817654 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:44.835294 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:44.844482 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:44.941658 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:44.967293 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:45.012383 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:45.100220 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:45.210708 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:45.338057 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:45.348922 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:45.413754 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:45.429210 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:45.434000 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:45.439980 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:45.447842 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:45.464975 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:45.536896 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:45.636368 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:45.637533 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:45.728636 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:45.756966 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:45.767791 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:45.819015 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:02:45.825249 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:45.860009 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:45.895773 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:45.925144 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:46.020416 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:46.064436 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:46.074508 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:46.134387 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:46.139356 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:46.140362 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:46.167025 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:02:46.214360 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 14:02:46.279894 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:46.281988 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:46.327335 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:46.331276 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:46.388494 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:46.406472 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:46.428627 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:46.479760 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 14:02:46.612927 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:46.623876 15414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:46.624701 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22eef55211ca4f65979022d2f4ae829f: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:46.810603 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:46.967209 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:46.981388 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:47.012938 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:47.088516 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:47.173980 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:47.294996 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:47.335366 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:47.427855 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:47.487864 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:47.546645 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:47.549432 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:47.619406 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:02:47.620116 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:47.675405 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:47.684433 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:47.715186 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:47.724787 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:47.829245 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:47.864068 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:47.929391 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:47.941567 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:47.965392 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:48.043697 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:48.124759 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:48.211087 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:48.214354 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:48.229849 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:48.281774 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:48.296064 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:48.313284 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:48.349349 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:48.443244 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:48.457193 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:48.468617 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:48.532384 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:02:48.543903 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:48.572901 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:48.577885 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:48.730321 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:48.767617 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:48.791249 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:48.795254 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:48.801206 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:48.863577 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:48.863998 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:48.878333 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:48.959664 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:49.227488 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:49.317404 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:49.321534 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:49.328506 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:49.522209 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:49.557935 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:49.763075 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:49.784657 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:49.806200 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:49.814563 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:49.817771 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:49.942940 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:49.952739 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:49.977883 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:50.075742 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:50.087750 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:50.106791 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:50.178083 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:50.208840 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:50.253496 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:50.315474 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:50.372891 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:50.379650 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:50.485702 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:50.514606 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:50.594566 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:50.673451 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:50.705559 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:50.723373 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:50.761406 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:50.779807 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:50.791365 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:50.829413 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:02:50.830085 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:50.899294 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:50.985844 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:50.986740 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:51.048774 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:51.119801 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:51.149998 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:02:51.171185 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:51.194192 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:51.199012 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:51.216912 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:51.239234 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 14:02:51.328390 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:51.373574 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:51.442648 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:51.442744 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:51.464007 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:51.705453 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 14:02:51.706079 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:51.706653 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:51.721738 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:51.750363 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:51.817324 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:51.915794 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:51.950004 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:52.263397 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:52.265625 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:52.321812 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:52.325131 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:52.351262 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:52.382961 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:52.428488 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:52.504915 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:52.716637 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:02:52.719453 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:52.719689 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:52.784909 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:52.795261 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:52.805855 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:52.882890 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:52.883632 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:52.914419 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:52.937896 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:53.012878 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:53.030274 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:53.103685 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:53.190995 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:53.278944 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:53.281203 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:53.328624 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:53.364339 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:53.412343 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:53.485440 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:53.500870 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:53.541502 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:53.646369 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:53.648119 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:02:53.675882 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:53.695684 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:53.737077 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:02:53.769263 24312 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:02:53.769415 24312 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:02:53.769874 24312 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:02:53.770946 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:53.771036 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:02:53.771117 15897 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:02:53.771263 24312 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:02:53.784385 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:53.791291 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:53.816399 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:53.829169 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:53.916769 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:53.931882 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:53.990962 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:54.068181 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:54.250773 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:54.327440 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:54.385880 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:54.438145 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:54.449352 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:54.514798 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:54.688557 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:54.818847 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:54.876340 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:54.909087 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:54.911288 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:55.061944 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:55.085918 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:55.146816 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:55.185871 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:55.427409 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:55.438982 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:55.449476 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:55.450959 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:55.466486 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:55.515525 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:55.540316 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:55.542552 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:55.574260 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:55.580350 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:55.581718 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:55.629689 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:55.692447 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:55.759820 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:55.809463 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:55.843461 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:02:55.859939 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:55.904408 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:55.975134 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:56.031411 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:56.112115 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:02:56.127319 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:56.147541 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:56.199957 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:56.237411 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:56.250255 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:56.254887 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:56.256690 15505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:02:56.303715 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:56.307619 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:56.312880 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:56.414836 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:56.540015 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:56.612445 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:56.703120 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:56.703713 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:56.766669 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 14:02:56.916610 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:56.954823 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:57.007258 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:57.158061 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:57.200302 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:57.287503 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:57.312752 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:57.382805 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:57.474707 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:57.494573 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:57.504515 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:57.573953 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:57.689364 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:57.722939 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:57.724478 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 14:02:57.725186 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:57.805191 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:57.813424 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:57.871014 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:57.893038 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:57.896860 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:57.923373 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:57.962896 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:58.063726 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:58.073689 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:58.141996 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:58.146392 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:58.149920 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:58.240061 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:58.366600 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:02:58.437930 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:58.438042 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:02:58.494647 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:58.507548 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:58.511483 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:58.532058 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:58.537161 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:58.538477 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:58.652382 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:58.807830 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:58.867000 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:02:58.911522 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:58.955117 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:59.039076 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:59.078156 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:59.174503 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:59.179821 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:02:59.197633 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:59.204880 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:02:59.462281 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:59.514959 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:02:59.565546 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:02:59.602640 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:02:59.611136 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:02:59.619213 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:02:59.706207 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:02:59.802016 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:02:59.999608 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:00.015405 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:03:00.067675 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:00.091666 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:00.107247 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:00.126948 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:00.184756 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:03:00.185200 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:00.194097 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:00.302968 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:03:00.314651 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:00.546559 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:00.574602 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:03:00.578585 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:00.579035 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:00.607882 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:00.646203 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:00.648685 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:03:00.705830 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:00.724531 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:00.788424 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:00.874127 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:03:00.874845 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:00.951728 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:03:00.990705 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:00.993151 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:01.019104 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:01.047745 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:01.068444 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:01.108683 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:01.193187 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:01.208735 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:01.379658 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:03:01.425540 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:01.436811 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:01.459287 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:01.461766 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:01.481962 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:01.596122 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:01.663455 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:01.751739 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:01.768368 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:01.777590 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 14:03:01.947793 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:02.004588 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:02.029060 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:02.067615 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:02.098433 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:02.180411 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:02.213035 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:02.220623 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:02.233345 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:02.332126 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:02.433946 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:02.467445 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:02.509315 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:02.580883 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:02.595939 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:02.604599 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:02.615442 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:02.702529 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:02.724838 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:02.766340 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:02.850191 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 14:03:02.904395 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:02.990250 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:03.068476 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:03.070485 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:03.124894 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:03.161202 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:03.172906 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:03.273183 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:03.321916 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:03.437937 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:03.464637 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:03.484048 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:03.599716 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:03.670047 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:03.690047 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:03.693764 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:03.708245 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:03.711398 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:03.725760 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:03.788664 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:03.839272 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:04.116048 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:04.116521 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:04.138118 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:04.156967 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:04.180233 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:04.226961 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:04.257803 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:04.289701 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:04.459429 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:04.492771 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:04.532553 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:04.593554 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:04.609545 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:04.642478 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:04.643232 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:04.699952 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:04.841879 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:04.904999 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:04.960978 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:05.006511 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:05.141446 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:05.172509 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:05.179695 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:05.211359 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:05.254606 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:05.311903 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:05.429939 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:05.454510 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:05.475282 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:05.627147 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:05.665187 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:05.715040 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:05.849856 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:05.874519 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:03:05.875205 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:05.881942 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:05.916836 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:05.917063 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:05.972208 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:05.975634 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:06.016760 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:06.017419 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:06.115177 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:06.138273 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:06.180639 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:06.188750 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:06.210280 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:06.228250 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:06.357795 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:06.398375 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240502 14:03:06.536473 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:06.593708 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:06.684690 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:06.719224 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:06.758450 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:06.774734 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:06.780544 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:06.834384 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:06.874655 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:06.880158 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:07.024382 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 14:03:07.024952 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:07.039146 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:07.076683 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:07.176105 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:07.245874 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:07.285373 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:07.286211 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:07.304069 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:07.359300 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:07.537863 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:07.663475 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:07.720919 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:07.747907 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:07.748116 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:07.753258 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:07.819044 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:07.834620 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:07.878046 24312 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:03:07.878190 24312 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:03:07.878711 24312 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:03:07.879182 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:03:07.879802 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:07.879962 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:03:07.880041 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:03:07.880210 24312 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:03:07.920471 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:07.922482 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:08.049322 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:08.222762 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:08.288719 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:08.374419 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:08.377988 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:08.405733 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:08.414340 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:08.453281 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:08.476889 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:08.489383 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:08.509851 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:08.515237 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:08.543342 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:08.557952 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:08.610601 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:08.639578 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:08.871836 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:08.916270 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:08.995608 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:09.051748 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:09.051748 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:09.110924 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:09.123826 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:09.161422 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:09.208827 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:09.242444 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:09.378901 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:09.478379 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:09.552351 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:09.576545 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:09.583629 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:09.676120 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:09.699357 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:09.779855 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:09.798842 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:09.811187 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:09.849778 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:10.052697 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:10.163163 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:10.261214 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:10.270368 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:10.286723 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:10.290724 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:10.294598 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:10.357734 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:10.367611 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:10.504703 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:10.577308 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:10.668110 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:10.754760 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:10.774942 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:10.787773 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:10.802102 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:10.808008 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:10.866109 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:10.902434 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 14:03:10.925843 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:11.019116 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:11.020275 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:11.071763 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:11.091488 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:11.104979 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:11.124801 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:11.159711 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:11.196691 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:11.404008 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:11.405086 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:03:11.405735 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:11.441665 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:11.446215 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:11.620215 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:11.630468 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:11.640430 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:11.641347 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:11.666836 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:11.711216 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:11.854467 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:11.929721 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:12.036804 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:12.093845 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:12.116182 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 14:03:12.116928 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:12.133981 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:12.158282 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:12.179296 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:12.204439 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:12.243881 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:12.552847 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:12.602787 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:12.618736 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:12.664906 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:12.688867 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:12.689796 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:12.749475 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:12.756708 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:12.858784 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:12.987174 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:12.987246 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:12.990201 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:13.037568 15898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 14:03:13.038352 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:13.121368 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:13.147979 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:13.150738 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:13.172163 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:13.176738 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:13.411638 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:13.427739 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:13.470180 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:13.549435 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:13.556648 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:13.563212 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:13.625062 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:13.676352 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:13.699615 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:13.729830 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:03:13.820749 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:13.916560 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:14.014472 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:14.059813 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:14.069209 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:14.072952 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:14.110803 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:14.154803 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:14.213205 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:14.276513 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:14.308827 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:14.317973 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:14.400410 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:14.445222 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:14.464735 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:14.556056 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:14.599417 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:14.641392 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:14.702417 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:14.967821 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:14.996726 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:15.048599 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:15.089279 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:15.097251 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:15.141463 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:15.156049 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:15.306171 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:15.340116 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:15.364089 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:15.405141 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:15.408003 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:15.411305 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:15.503762 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:15.559468 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:15.562331 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:15.619045 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:15.655167 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:15.771981 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:15.886036 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:15.912431 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:15.927464 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:03:15.981467 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:16.020421 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:16.088279 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:16.106595 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:16.145867 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:16.205256 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:16.306948 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:16.310734 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:16.328800 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:03:16.405982 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240502 14:03:16.406715 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:16.416360 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:16.477300 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:16.477417 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:16.491613 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:16.512516 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:16.603895 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:16.613546 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:16.640895 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:16.709964 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:16.772465 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:16.806803 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:16.898636 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:17.057142 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:17.145154 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 14:03:17.162989 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:17.195252 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:17.199910 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:17.245899 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:17.290511 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:17.322883 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:17.388509 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:17.446630 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:03:17.563212 24866 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:03:17.563383 24866 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:03:17.563952 24866 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:03:17.564956 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:17.565217 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:03:17.565292 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:03:17.565452 24866 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:03:17.670678 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:17.707599 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:17.743561 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:17.752177 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:17.754056 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:17.774966 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:17.820537 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:17.827488 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:17.846479 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:17.954341 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:17.955626 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:18.091418 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:18.124435 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:18.146345 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:18.245322 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:18.247678 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:18.325239 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 55 similar messages]
W20240502 14:03:18.349200 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:18.359818 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:18.441860 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:18.483796 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:18.555246 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:18.578553 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:18.601697 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:18.753342 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:18.837853 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:18.852613 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:18.857347 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:18.859863 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:18.867192 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:18.894316 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:03:19.079341 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:19.106657 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:19.166908 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:19.231981 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:19.238123 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:19.267171 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:19.321014 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:19.332363 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:19.357352 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:19.454638 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:03:19.658653 25124 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:03:19.658805 25124 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:03:19.659258 25124 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:03:19.660293 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:19.660390 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:03:19.660470 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:03:19.660591 25124 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:03:19.660806 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:19.687744 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:19.691622 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:19.725288 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:19.945039 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:20.001773 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:20.020515 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:20.029227 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:20.029273 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:20.034366 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:20.074370 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:20.091773 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:20.109962 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:20.426014 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:20.431528 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:20.458060 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:20.464159 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:20.467099 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:20.498239 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:20.515689 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:20.544917 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:20.785391 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:20.826105 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:20.962651 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:20.967636 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:03:20.968209 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:20.982393 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:20.990103 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:20.994776 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:20.998370 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:21.020349 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:21.071084 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:21.164017 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:21.334275 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:21.365036 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:21.447477 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:21.450430 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240502 14:03:21.451010 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:21.454924 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:03:21.516310 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:21.521385 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:21.546546 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:21.552995 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:21.736585 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:21.765434 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:21.792048 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:21.923118 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:21.929776 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:21.969316 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:22.018586 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:22.051959 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:22.090677 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:22.160080 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:03:22.258203 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:22.284430 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:22.292829 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:22.434638 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:22.562392 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:22.565026 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:22.566968 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:22.600807 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:22.708782 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:22.724731 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:22.727144 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:22.861522 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:22.929949 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:22.963740 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:23.037020 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:23.109723 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:23.153236 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:23.175467 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:23.213608 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:23.232177 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:23.313978 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:23.324966 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:23.383805 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:23.390476 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 14:03:23.423382 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:23.485498 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:23.497504 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:23.561578 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:23.627682 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:23.723785 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:23.786355 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:23.902238 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:23.913180 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:23.933950 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:03:23.957890 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:24.077227 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:24.096006 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:24.193718 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:24.197212 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:24.219594 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:24.272418 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:24.353390 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:24.411718 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:24.489439 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:24.499653 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:24.504406 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:24.518155 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:24.555961 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:24.616650 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:24.696791 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:24.777158 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:24.935737 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:24.986717 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:25.004457 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:25.022218 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:25.110710 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:25.125535 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:25.176901 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:25.250692 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:25.252372 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:25.265542 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:25.441381 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:25.466610 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:25.604665 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:25.651378 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:25.663331 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:25.732853 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:25.763546 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:25.764420 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:25.847080 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:25.851357 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:25.874332 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:25.889806 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:25.941319 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:25.950726 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:25.993894 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 14:03:26.049257 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:26.064213 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:26.163079 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:26.164831 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:26.322077 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:26.327107 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:26.487088 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:26.496505 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:26.523051 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:03:26.534081 15505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:03:26.563172 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:26.576586 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:26.582408 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:26.616237 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:26.692986 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:26.863415 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:26.889335 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:26.965950 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:26.997572 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:27.005491 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:27.012881 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:27.019819 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:27.031283 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:27.060487 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:27.124154 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:27.235828 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:27.284334 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:03:27.414808 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:27.449290 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:27.474570 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:27.544283 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:27.547734 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:27.633414 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:27.708243 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:27.802650 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:27.901204 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:27.908474 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:27.973078 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:27.976241 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:27.996430 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:28.028100 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:28.131604 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:28.137578 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:28.190336 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:28.262331 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:28.284370 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:28.287108 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:28.397652 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:28.402459 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:28.452105 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:28.473457 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:28.487210 15898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:03:28.487820 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:28.565744 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:28.581079 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:28.780192 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:28.811681 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:28.887079 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:28.889868 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:03:28.896458 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:28.905166 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:28.911242 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:29.003697 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:29.108819 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:29.215862 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:29.235432 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:29.237962 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:29.276757 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:29.320062 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:29.353158 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:29.373353 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:29.439658 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:29.518867 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:29.641911 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:29.803656 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:29.840327 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:29.883517 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:29.925623 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:29.971995 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:29.977927 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:29.980965 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:29.981200 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:30.110268 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:30.166256 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:30.177788 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:30.315049 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:30.446669 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:30.482522 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:30.519726 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:30.534729 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:30.567128 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:30.572379 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:30.580993 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:30.582363 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:30.595110 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:30.741755 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:30.742254 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:30.854588 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:30.938536 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:30.972795 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:31.004313 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:31.007450 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:03:31.008026 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:31.109149 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:31.251060 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:31.275450 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:31.275763 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:31.280426 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:31.300158 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:31.330370 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:03:31.426997 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:31.502647 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:31.516959 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:31.548501 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:03:31.614846 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:31.787441 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:31.912927 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:31.932896 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:31.984063 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:32.031622 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:32.077847 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:32.079571 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:32.083948 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:32.159148 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:32.195060 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:32.216305 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:32.277076 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:32.278385 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:32.387518 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 14:03:32.397008 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:32.506997 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:32.508889 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:32.508875 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:32.570346 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:32.578126 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:32.676676 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:32.704427 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:32.834131 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:32.940289 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:32.948200 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:32.984205 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:33.032186 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:33.055246 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:33.083413 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:33.097283 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:33.123584 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:33.253062 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:33.291893 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:33.294847 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:33.386974 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:33.420157 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:33.463239 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:33.498339 15898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 14:03:33.498929 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:33.503890 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:33.539088 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:33.593097 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:33.843000 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:33.843557 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:33.847510 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:03:33.857299 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:33.910255 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:33.971665 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:34.040836 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:34.061781 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:34.378927 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:34.404204 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:34.409781 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:34.428843 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:34.456459 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:34.463268 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:34.498894 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:34.561115 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:34.675534 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:34.687726 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:34.715854 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:34.718768 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:34.901286 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:35.055613 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:35.065459 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:35.087251 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:35.098122 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:35.101997 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:35.141991 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:35.143694 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:35.213892 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:35.225276 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:35.329095 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:35.347685 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:35.397249 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:35.538709 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:35.613412 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:35.624538 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:35.626205 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:35.697783 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:35.709846 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:35.735627 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:35.807582 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:35.814992 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:35.985509 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:36.092422 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:36.101450 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:03:36.168015 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:36.172636 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:36.174124 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:36.205092 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:36.217381 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:03:36.293061 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:36.293056 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:36.311771 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:36.324215 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:36.424768 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:36.605334 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:36.616281 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:03:36.711694 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:36.783566 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:36.801107 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:36.816941 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:36.875555 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:36.972239 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:37.023741 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:37.031648 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:37.074565 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:37.090996 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:37.103327 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:37.196383 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:37.221647 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:37.283403 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:37.349905 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:37.363692 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:37.391920 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 14:03:37.423991 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:37.484318 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:37.563467 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:37.681354 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:37.746379 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:37.749482 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:37.819782 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:37.820647 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:37.871394 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:37.934211 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:37.993840 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:38.001325 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:38.033907 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:38.060276 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:38.083170 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:38.174335 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:38.246218 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:38.257398 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:38.314745 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:38.432665 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:38.432664 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:38.461328 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:38.601675 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 14:03:38.602419 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:38.611388 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:38.632814 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:03:38.682801 25482 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:03:38.682940 25482 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:03:38.683334 25482 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:03:38.684278 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:38.684363 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:03:38.684448 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:03:38.684623 25482 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:03:38.701342 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:38.739600 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:38.772246 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:38.799719 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:38.809710 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:03:38.951973 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:38.969334 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:39.143721 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:39.217752 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:39.249238 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:39.292515 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:39.444406 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:39.468794 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:39.482048 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:39.562352 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:39.594563 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:39.600227 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:39.638366 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:39.671779 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:39.683136 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:39.710106 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:39.743867 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:39.796751 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:39.847299 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:40.036504 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:40.091678 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:40.133708 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:40.140691 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:40.194953 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:40.207957 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:40.222579 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:40.297961 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:40.339272 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:40.453665 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:40.484799 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:40.498798 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:40.515857 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:40.522352 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:40.545038 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:40.603282 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:40.676095 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:40.768752 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:40.951596 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:40.959139 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:41.014598 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:41.028725 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:41.072335 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:41.121244 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:41.132988 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:41.152738 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240502 14:03:41.153427 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:41.202502 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:41.203979 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:41.258183 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:41.358557 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:03:41.450127 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:41.552527 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:41.564416 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:41.637323 15505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:03:41.726538 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:41.802008 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:41.819697 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:41.944802 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:41.979142 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:41.995815 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:42.025168 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:42.052280 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:42.152109 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:42.157994 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:42.238118 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:42.323371 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:42.332976 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:42.400353 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 14:03:42.406960 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:42.450874 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:42.512457 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:42.543609 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:42.579672 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:42.589088 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:42.799268 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:42.807884 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:42.825369 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:42.825479 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:42.834437 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:42.871490 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:42.940120 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:42.944901 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:42.970340 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:42.998170 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:43.004541 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:43.027483 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:43.064898 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:43.309852 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:43.328253 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:43.415468 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:43.419467 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:43.432049 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:43.565379 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:43.565454 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:43.660902 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:43.690546 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:43.703925 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:43.734484 15898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 14:03:43.735088 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:43.748342 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:43.759585 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:43.921224 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:44.026067 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:03:44.030460 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:44.182085 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:44.196132 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:44.306413 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:44.320585 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:44.340243 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:44.387365 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:44.442922 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:44.454890 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:44.594766 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:44.610090 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:44.613533 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:44.690649 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:44.741837 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:44.833364 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:44.857187 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:44.894498 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:44.936606 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:45.140637 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:45.144042 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:45.178194 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:45.267549 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:45.311216 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:45.319289 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:45.376116 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:45.391235 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:45.401588 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:45.433938 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:45.452157 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:45.457711 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:45.491575 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:45.562882 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:45.587792 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:45.616830 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:45.716470 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:45.717347 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:45.731909 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:45.790086 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:46.015007 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:46.069005 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:46.119907 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:46.137818 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:46.178551 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:46.181303 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:03:46.189028 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:46.243844 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:46.280469 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:46.336910 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:03:46.505420 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:46.547904 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:46.634881 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:46.639357 15414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:46.640064 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22eef55211ca4f65979022d2f4ae829f: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:46.688550 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:03:46.689178 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:46.737675 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:46.785763 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:46.876206 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:47.092839 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:47.099416 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:47.142740 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:47.190101 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:47.201516 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:47.242594 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:47.273061 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:47.306438 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:47.467317 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:47.491475 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 14:03:47.518250 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:47.534523 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:47.559810 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:47.682273 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:47.735849 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:47.768076 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:47.778431 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:47.817320 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:47.820940 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:47.823796 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:47.842803 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:47.879540 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:47.929776 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:47.932607 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:47.940158 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:48.076103 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:48.134341 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:48.191870 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:48.221526 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:48.228978 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:48.247661 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:48.286357 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:48.325898 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:48.344250 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:48.459117 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:48.480072 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:48.708642 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:48.717545 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:48.750593 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 14:03:48.812871 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:48.841971 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:48.853788 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:49.008100 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:49.060755 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:49.086488 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:03:49.103533 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:49.137306 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:49.207742 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:49.296494 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:49.507930 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:49.559546 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:49.597416 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:49.633716 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:49.648392 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:49.729454 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:49.759030 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:49.872601 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:49.906234 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:50.040401 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:50.094071 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:50.127646 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:50.183795 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:50.188006 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:50.229389 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:50.265158 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:50.298205 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:50.298458 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:50.321018 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:50.326814 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:50.377686 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:50.398699 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:50.406255 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:50.430472 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:50.524868 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:50.612545 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:50.616520 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:50.650903 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:50.716575 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:50.718441 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:50.806370 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:50.821131 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:50.867482 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:50.895022 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:51.053260 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:51.214993 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:03:51.215584 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:51.243440 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:51.247898 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:51.257102 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:51.260185 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:51.265359 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:51.312664 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:51.500847 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:51.625599 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:03:51.645110 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:51.698937 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:03:51.737720 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:51.746469 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:51.747360 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:51.814203 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:52.087934 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:52.152089 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:52.188098 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:52.198431 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:52.204097 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:52.232983 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:52.434834 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:52.494067 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:03:52.504742 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:52.559005 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:52.621047 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:52.637496 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:52.641789 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:52.666491 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:52.673548 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:52.720772 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:52.732316 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:52.811367 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:52.811931 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:52.920571 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:52.941270 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:52.967003 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:52.979007 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:52.999258 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:53.109113 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:53.134670 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:53.143325 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:53.166186 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:53.219554 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:53.220939 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:53.328253 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:53.329383 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:53.345456 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:53.360411 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:53.427825 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:53.644872 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:53.697726 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:53.714887 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:53.747584 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
I20240502 14:03:53.750684 25997 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:03:53.750815 25997 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:03:53.751231 25997 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:03:53.751572 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 14:03:53.752171 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:53.752300 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:03:53.752367 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:03:53.752494 25997 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:03:53.757655 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:53.820894 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:53.863168 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:54.004225 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:54.007364 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:54.085399 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:54.113488 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:03:54.262132 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:54.264273 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:54.285207 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:54.532687 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:54.601509 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:54.678730 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:54.682097 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:54.721410 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:54.788247 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:54.936151 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:54.990355 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:55.001194 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:55.009878 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:55.147699 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:55.163944 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:55.166061 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:55.172850 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:55.195045 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:55.264822 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:55.277213 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:55.346587 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:55.399153 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:55.414189 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:55.482946 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:55.508801 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:55.522765 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:55.525444 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:55.587754 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:55.662252 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:55.686141 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:55.711864 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:55.774354 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:55.815505 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:55.903316 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:55.935457 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:55.973416 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:56.003036 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:56.012447 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:56.146219 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:56.262826 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:56.268910 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:56.281133 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:56.297389 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:03:56.371255 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:56.457605 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:56.463934 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:03:56.531757 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:56.533641 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:56.541527 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:56.620545 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:56.628209 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:56.726691 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:03:56.838510 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:56.958041 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:56.973080 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:57.034862 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:57.034989 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:57.093346 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:57.384793 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:57.416867 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:57.426265 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:57.530460 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:03:57.581411 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:57.603737 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:57.611032 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:57.664315 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:57.664492 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 14:03:57.703063 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:57.733356 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:57.757458 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:57.780490 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:57.796123 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:57.867776 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:57.889315 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:57.902840 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:57.982481 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:58.001529 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:58.002674 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:58.083969 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:58.180209 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:58.309566 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:58.314365 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:58.371282 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:58.413357 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:58.422596 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:58.438947 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:03:58.455691 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:58.506646 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:58.521723 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:58.586118 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:58.600018 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:03:58.713312 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:03:58.832548 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:03:58.902140 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:03:58.906682 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:58.995760 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:59.006475 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:59.082927 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:59.089270 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:59.100768 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:59.193265 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:59.205695 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:03:59.401859 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:03:59.406677 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:59.536352 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:03:59.664456 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:03:59.697603 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:03:59.907521 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:03:59.909289 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:03:59.997256 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:04:00.028832 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:00.058914 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:00.063947 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:00.135720 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:00.151338 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:04:00.168359 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:00.177058 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:04:00.250043 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:00.250852 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:00.277906 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:04:00.325954 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:00.376080 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:00.481824 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:00.527096 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:00.554358 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:04:00.582823 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:00.630602 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:00.634845 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:04:00.691147 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:04:00.749172 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:00.799300 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:00.823635 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:00.841694 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:00.895884 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:00.947582 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:01.011268 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:01.014225 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:01.027781 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:01.035099 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:01.070551 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:01.111573 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:04:01.273651 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:01.331737 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 14:04:01.387626 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:01.391420 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:01.462761 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:01.486119 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:01.489853 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:01.531628 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:01.706584 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:01.766803 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240502 14:04:01.767424 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:01.776377 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:01.867116 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:01.916083 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:01.954011 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:02.159754 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:02.166922 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:02.239961 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:02.284672 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:02.414326 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:02.446609 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:02.584205 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:02.588619 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:02.599787 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:02.700353 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 14:04:02.701066 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:02.729676 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:02.750536 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:02.781795 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:02.800884 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:02.819703 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:02.925438 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:02.926694 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:02.989765 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:03.034710 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:03.069111 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:03.070812 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:03.118496 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:03.150918 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:03.158134 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:03.195927 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:03.196007 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:03.249030 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:03.472229 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:03.534173 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:03.556655 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:03.575348 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:03.590211 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:03.666747 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:03.679714 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:03.724105 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:03.748034 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:03.758009 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:03.775856 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:03.904402 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:03.910729 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:04:03.911412 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:04.027279 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:04.049412 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:04.156733 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:04.305323 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:04.346702 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:04.406787 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:04.535308 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:04.620138 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:04.632490 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:04.655325 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:04.767190 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:04.813198 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:05.008738 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:05.011638 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:05.120150 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:05.139209 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:05.142314 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:05.321460 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:05.336903 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:05.363241 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:05.369091 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:05.404875 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:05.411872 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:05.425663 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:05.429448 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:05.436293 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:05.475389 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:05.499140 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:05.584782 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:05.606110 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:05.646114 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:05.739988 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:05.757766 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:05.796171 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:05.837381 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:05.865119 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:05.930534 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:05.975215 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:05.978075 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:05.982990 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:04:05.982965 26415 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:05.983129 26415 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:04:05.983464 26415 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:04:05.984305 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:05.984390 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:04:05.984447 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:04:05.984581 26415 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:04:06.090734 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:06.161041 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:06.187170 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:06.201761 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:06.215667 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:06.307087 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:06.371039 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:06.373998 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:06.383440 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:04:06.409751 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:06.526724 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:06.535144 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:06.617544 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:06.810458 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 14:04:06.914018 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:06.942456 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:07.025698 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:07.027958 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:07.123440 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:07.309944 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:07.404000 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:07.470935 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:07.513743 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:07.598165 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:07.600396 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:07.620208 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:07.650224 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:07.672654 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:07.759229 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:07.762019 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 14:04:07.766659 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:07.815081 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:07.834218 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:07.863523 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:07.995918 26415 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:07.996069 26415 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:04:07.996498 26415 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:04:07.997486 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:07.997750 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:04:07.997823 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:04:07.997931 26415 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:04:08.055369 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:08.072450 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:08.131421 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:08.146659 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:08.164662 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:08.180142 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:08.230909 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:08.278584 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:08.300293 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:08.325024 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:08.330765 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:08.415820 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:08.429208 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:08.494817 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:08.507472 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:08.563464 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:08.595448 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:08.642634 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:08.730882 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:08.811013 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:08.840770 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:08.873598 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:08.945724 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:08.950616 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240502 14:04:08.951165 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:08.956415 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:09.002566 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:09.073369 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:09.414012 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:09.435340 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:09.438230 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:09.492563 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:09.636703 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:09.665397 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:09.708858 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:09.816452 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:09.945016 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:09.947412 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:10.089280 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:10.103255 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:10.160234 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:10.162248 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:10.195875 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:10.204631 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:10.366016 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:10.380254 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:10.427274 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:10.453704 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:10.473667 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:10.537758 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:10.586835 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:10.633293 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:10.636622 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:10.660356 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:10.684908 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:10.708309 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:10.757678 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:10.759660 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:10.808089 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:10.814713 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:10.906159 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:10.963022 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:11.002655 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:11.010782 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:11.043237 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:11.093616 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:11.105871 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:11.127197 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:11.217922 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:11.299383 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:11.392496 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240502 14:04:11.392853 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:11.418361 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:11.454660 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:11.511016 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:11.639866 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:11.643783 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:11.818531 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:11.828338 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240502 14:04:11.887403 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:11.983497 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:12.076817 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:12.161026 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:12.189356 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:12.210590 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:12.329885 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:12.379568 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:12.437800 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:12.558436 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:12.599828 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:12.638259 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:12.659719 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:12.660051 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:12.737071 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:12.816123 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:12.819275 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 14:04:12.819872 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:12.895076 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:12.944520 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:12.953603 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:12.959800 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:12.983031 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:13.136437 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:13.169175 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:13.173611 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:13.191653 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:13.195315 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:13.280030 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:13.284636 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:13.355984 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:13.368727 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:13.378799 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:13.388006 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:13.441454 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:13.498317 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:13.533061 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:13.536748 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:13.635787 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:13.672106 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:13.672602 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:13.796617 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:13.813637 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:13.894192 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:13.953078 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:04:13.986088 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:04:13.986748 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:14.039116 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:14.069597 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:14.296720 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:14.419358 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:14.444685 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:14.469147 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:14.590746 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:14.633759 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:14.823972 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:04:14.864624 26569 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:14.864743 26569 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:04:14.865128 26569 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:04:14.866148 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:14.868685 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:04:14.868763 15897 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:04:14.868934 26569 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:04:14.876777 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:14.995340 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:15.008132 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:15.039191 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:15.106580 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:15.111455 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:15.140448 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:15.201786 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:15.293965 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:15.298977 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:15.300164 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:15.315757 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:15.354035 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:15.400005 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:15.408020 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:15.447539 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:15.562454 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:15.571638 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:15.603772 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:15.659341 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:15.678628 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:15.688759 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:15.711395 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:15.782371 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:15.800405 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:15.816021 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:15.834348 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:15.839435 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:16.007561 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:16.031497 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:16.052299 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:16.149695 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:16.209105 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:04:16.251593 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:16.254338 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:16.300273 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:16.309926 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:16.390811 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:16.460294 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:04:16.537446 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:16.653208 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:16.722790 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:16.766218 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:16.798396 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:16.841928 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 14:04:16.971863 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:16.990233 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:17.150097 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:17.162201 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:17.336824 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:17.394570 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:17.571771 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:17.628640 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:17.662147 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:17.692487 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:17.722121 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:17.829743 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:17.839936 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:17.865633 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 14:04:17.866319 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:17.880339 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:17.959872 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:17.979422 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:17.985595 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:18.048571 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:18.072742 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:18.130136 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:18.147624 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:18.172169 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:18.176800 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:18.192818 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:18.197225 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:18.289599 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:18.292840 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:18.305584 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:18.365300 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:18.411319 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:18.420897 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:18.436947 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:18.575042 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:18.577950 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:18.599514 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:18.729650 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:18.744624 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:04:18.748703 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:18.778059 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:18.823500 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:18.873622 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:18.905792 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:18.988727 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:19.128876 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 14:04:19.149194 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:19.165236 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:19.187222 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:19.334416 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:19.527813 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:19.550449 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:19.572997 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:19.581696 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:19.784222 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:19.998167 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:20.108008 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:20.125123 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:20.194097 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:20.254788 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:20.263168 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:20.293798 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:20.333956 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:20.431725 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:20.435812 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:20.441764 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:20.506996 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:20.524238 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:20.588928 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:20.670516 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:20.676528 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:20.677626 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:20.683663 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:20.694952 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:20.711783 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:20.712103 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:20.719218 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:20.719594 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:20.808797 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:20.884593 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:20.888801 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:20.936861 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:20.975777 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:21.009646 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:21.015566 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:21.036135 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:21.079672 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:21.085969 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:21.230950 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:21.250473 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:21.311591 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:04:21.376947 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:21.490597 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:21.509270 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 115 similar messages]
W20240502 14:04:21.565603 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:21.638181 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:21.696131 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:21.864660 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:04:21.865355 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:21.868882 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:21.912289 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:22.022377 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:22.071223 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:22.188136 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:22.318624 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:22.333402 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:22.449204 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:22.516022 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:22.774158 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:22.774492 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:22.788928 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:22.815500 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:22.826727 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:22.829258 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:22.884932 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 14:04:22.895038 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:23.007263 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:23.058773 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:23.060462 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:23.098675 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:23.133109 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:23.166653 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:23.198251 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:23.228003 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:23.243778 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:23.262246 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:23.319321 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:23.325431 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:23.330879 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:23.364655 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:23.367727 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:23.375643 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:23.398422 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:23.446841 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:23.680444 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:04:23.702986 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:23.721598 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:23.727870 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:23.772455 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:23.778450 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:23.808360 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:23.863107 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:23.874742 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:23.889596 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:23.895536 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:24.157450 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:24.272012 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:24.373004 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:24.389720 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:04:24.411926 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:24.444722 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:24.512509 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:24.602967 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:24.654211 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:24.656504 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:24.669927 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:24.842145 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:24.913159 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:25.061467 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:25.185364 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:25.275949 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:25.294942 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:25.301323 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:25.392673 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:25.401407 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:25.451746 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:25.572569 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:25.606587 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:25.616124 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:25.621536 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:25.627552 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:25.632835 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:25.646358 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:25.661332 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:25.705454 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:25.730087 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:25.754930 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:25.802567 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:25.878438 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:25.935528 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:25.961978 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:26.026916 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:26.034436 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:26.063633 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:26.106253 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:26.109694 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:04:26.130177 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:26.143383 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:26.173174 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:26.267364 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:26.272666 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:26.380537 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:26.467800 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:26.483462 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:26.550166 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:04:26.551900 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:26.560154 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:26.729857 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:26.779635 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:26.915669 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:26.948469 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:04:26.952531 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:27.037056 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:27.108198 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:27.129428 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:27.134569 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:27.178704 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:27.498467 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:27.513506 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:27.562351 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:27.615224 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:27.619518 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:27.741015 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:27.810516 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:27.886006 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 14:04:27.938632 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:27.954977 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:27.974684 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:28.009999 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:28.064252 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:28.108255 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:28.150940 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:28.152370 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:28.170850 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:28.187834 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:28.209847 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:28.311815 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:28.335975 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:28.388834 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:28.399483 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:28.399580 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:28.410092 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:28.461064 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:28.481288 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:28.485972 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:28.495831 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:28.516055 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:28.530789 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:28.602521 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:04:28.635493 26976 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:28.635628 26976 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:04:28.636049 26976 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:04:28.637100 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:28.637187 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:04:28.637248 15897 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:04:28.637387 26976 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:04:28.680820 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:28.688756 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:28.764596 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:28.984781 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:04:28.989156 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:29.051520 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:29.084446 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:29.091037 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:29.148242 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:29.220306 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:29.358132 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:29.418344 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 14:04:29.463088 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:29.465764 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:29.477226 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:29.521924 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:29.585906 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:29.695876 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:29.843420 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:30.035086 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:30.042745 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:30.070353 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:30.079358 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:30.161947 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:30.214267 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:30.332198 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:30.350253 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:30.375525 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:30.458683 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:30.503327 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:30.519286 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:30.533807 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:30.571683 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:30.593796 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:30.626318 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:30.638890 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:30.651588 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:30.686269 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:30.730342 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:30.790930 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:30.824565 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:30.857380 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:30.864544 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:30.912179 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:31.019166 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:31.027819 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:31.076301 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:31.180088 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:31.184549 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:31.185707 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:31.266839 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:31.285789 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:31.416918 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:31.461129 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:31.546658 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:31.554932 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:31.583103 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 14:04:31.584124 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:04:31.727288 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:31.752517 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:31.795431 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:31.865810 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:31.881654 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:32.010454 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:32.030822 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240502 14:04:32.200131 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:32.245842 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:32.299662 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:32.556993 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:32.558085 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:32.601740 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:32.646755 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:32.677521 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:32.711045 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:32.744143 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:32.813346 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:32.898110 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:32.917860 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 14:04:32.918581 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:32.929459 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:33.039539 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:33.047343 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:33.054154 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:33.090267 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:33.150265 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:33.158032 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:33.170365 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:33.187079 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:33.190717 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:33.250025 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:33.323314 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:33.345049 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:33.377290 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:33.496152 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:33.505041 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:33.575132 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:33.639087 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:33.665762 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:33.690241 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:33.697327 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:33.704550 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:33.755721 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:33.757663 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:33.828436 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:33.963330 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:34.057934 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:34.085363 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:34.104084 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:34.125550 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:34.286851 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:34.286947 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:04:34.473840 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:04:34.474520 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:34.545291 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:34.668959 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:34.718199 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:34.732686 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:35.015614 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:35.085120 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:35.089466 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:35.103304 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:35.209904 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:35.278868 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:35.288309 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:35.290971 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:35.334079 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:35.346640 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:35.392352 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:35.492204 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:35.514597 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:35.539288 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:35.640132 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:35.675438 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:35.675776 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:35.704139 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:35.721998 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:35.758582 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:35.817198 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:35.823585 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:35.844569 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:35.935501 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:35.959044 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:35.990919 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:36.020380 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:36.134692 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:36.145649 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:36.190706 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:36.192049 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:36.216503 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:36.258764 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:36.335896 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:36.354879 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:36.437454 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:36.447136 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:36.570883 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:36.591476 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:36.609177 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240502 14:04:36.672552 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:36.743306 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:36.746840 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:04:36.901710 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:36.996635 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:37.096869 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 14:04:37.104310 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:37.159927 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:37.267932 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:37.283783 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:37.286525 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:37.400192 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:37.413589 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:37.430694 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:37.610867 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:37.647857 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:37.668542 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:37.733706 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:37.748312 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:37.749936 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:37.805274 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:37.828191 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:37.948292 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 14:04:37.948907 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:38.009618 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:38.010093 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:38.162575 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:38.164462 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:38.169533 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:38.176632 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:38.217023 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:38.245137 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:38.310364 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:38.325786 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:38.360484 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:38.429942 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:38.430349 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:38.438174 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:38.493213 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:38.502725 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:38.561401 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:38.734755 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:38.736446 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:38.799829 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:38.804203 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:38.833568 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:38.853498 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:38.919358 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:39.050719 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:39.071220 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:39.108179 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:39.321264 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:04:39.350075 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:39.396485 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:39.403365 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:39.428095 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:39.547689 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:04:39.548307 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:39.790644 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:39.817385 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:39.852903 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:39.937994 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:39.994603 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:40.111840 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:40.164969 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:40.181216 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:40.198935 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:40.229867 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:40.286644 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:40.331671 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:40.387311 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:40.481007 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:40.538437 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:40.590504 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:40.624543 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:40.631374 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:40.671783 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:40.679111 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:40.707865 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:40.710130 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:40.745096 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:40.769193 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:40.813829 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:40.819187 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:40.861955 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:41.033217 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:41.083350 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:41.103055 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:41.154783 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:41.171494 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:41.213111 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:41.286990 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:41.300868 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:41.344110 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:41.379539 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:41.380940 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:41.591667 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:41.619670 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:41.625830 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:04:41.643113 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:41.658399 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:41.670758 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:04:41.702239 27233 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:41.702389 27233 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:04:41.702816 27233 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:04:41.703754 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:41.703840 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:04:41.703898 15897 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:04:41.704048 27233 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:04:41.784410 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:41.803715 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:41.930493 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:04:42.041214 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:42.115097 15505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240502 14:04:42.190351 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:42.243780 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:42.244413 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:42.300436 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:42.314105 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:42.324141 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:42.443625 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:42.482733 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:42.586061 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:42.626920 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:42.699920 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:42.732856 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:42.742009 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:42.749681 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:42.806983 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:42.925431 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:43.017838 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:43.026557 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 14:04:43.029323 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:43.034081 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:43.043599 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:43.120739 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:43.145781 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:43.180665 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:43.223052 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:43.232698 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:43.244884 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:43.324952 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:43.328024 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:43.368248 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:43.375221 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:43.534770 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:43.567726 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:43.632336 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:43.665639 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:43.742755 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:43.770654 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:43.811097 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:43.838876 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:44.004360 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:44.030053 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:44.043040 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:44.184988 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:44.194620 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:44.197400 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:44.222172 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:44.226511 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:44.435801 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:44.510489 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:04:44.572057 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:44.688133 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 14:04:44.695183 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:44.741348 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:44.787981 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:44.898470 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:44.973698 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:44.994915 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:45.014923 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:45.027331 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:45.044294 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:45.067785 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:45.201915 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:45.210506 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:45.224486 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:45.255182 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:45.343034 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:45.407444 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:45.431088 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:45.500916 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:45.594583 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:45.640125 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:45.716804 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:45.724577 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:45.743187 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:45.774477 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:45.841022 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:45.858271 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:45.861711 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:45.909797 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:45.939539 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:45.971742 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:45.976008 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:45.986742 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:46.130985 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:46.210275 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:46.255507 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:46.264430 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:46.288163 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:46.428948 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:46.502233 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:46.510807 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:46.601339 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:46.667726 15414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:46.668455 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22eef55211ca4f65979022d2f4ae829f: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:46.707834 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:04:46.736491 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:04:46.740608 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:46.758164 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:46.771832 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:46.825773 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:47.056979 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:47.069694 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:47.085510 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:47.137838 15505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:04:47.155362 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:47.361024 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:47.461776 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:47.482930 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:47.510589 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:47.567672 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:47.578470 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:47.605233 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:47.621682 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:47.637930 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:47.759370 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:47.811314 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:47.814349 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:47.912554 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:47.926980 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:48.015616 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:48.048417 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 14:04:48.073021 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:48.079222 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:48.144753 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:48.175086 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:48.217797 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:48.277037 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:48.304698 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:48.431596 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:48.436054 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:48.487864 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:48.542748 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:48.593727 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:48.614324 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:48.656833 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:48.660742 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:48.689802 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:48.696894 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:48.703470 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:48.748214 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:48.813637 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:48.910251 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:48.953117 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:48.996870 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:49.027809 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:49.067839 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:49.184983 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:04:49.194464 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:49.286191 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:49.381654 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:49.473423 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:49.591161 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:49.626403 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:49.639511 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:49.675218 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:49.774791 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:49.792752 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:50.007401 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:50.008096 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 14:04:50.008648 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:50.013659 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:50.068797 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:50.076835 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:50.082438 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:50.143294 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:50.237962 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:50.257798 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:50.345217 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:50.442615 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:50.499699 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:50.525874 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:50.558104 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:50.568459 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:50.788292 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:50.825194 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:50.861681 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:50.865962 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:50.886118 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:50.910092 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:50.935851 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:51.012712 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:51.047345 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:51.096498 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:51.111699 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:51.178079 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:51.222141 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:51.276654 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:51.278481 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:51.289064 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:51.309067 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:51.343008 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:51.376834 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:51.478128 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:51.481359 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:51.488255 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:51.494030 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:51.693014 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:04:51.721364 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 14:04:51.732156 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:51.790946 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:51.854997 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:51.927191 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:51.934886 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:52.098451 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:52.144625 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:04:52.146783 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:52.183310 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:52.198257 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:52.416100 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:52.477946 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:52.518087 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:52.532887 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:52.581892 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:52.595283 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:52.603387 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:52.665057 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:52.831097 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:52.937975 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:52.938385 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:53.052465 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:53.066376 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 14:04:53.101375 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:53.134032 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:53.218755 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:53.226106 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:53.228199 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:53.242832 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:53.289129 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:53.296253 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:53.320894 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:53.338011 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:53.406458 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:53.440933 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:53.504971 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:53.605954 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:53.636246 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:53.697115 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:53.733913 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:53.775692 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:53.812144 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:53.827167 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:53.866060 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:53.867318 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:53.897289 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:53.943526 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:54.054471 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:54.095950 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:04:54.096964 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:54.122059 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:54.125885 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:54.305433 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:54.316695 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:54.556334 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:54.618256 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:54.652240 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:54.654595 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:54.699908 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:54.854166 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:54.951740 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:54.968310 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:55.043465 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:55.074725 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 14:04:55.075443 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:55.187866 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:55.230221 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:55.233867 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:55.271728 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:55.377683 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:55.590505 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:55.601788 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:55.640270 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:55.640889 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:55.691627 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:55.695183 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:55.738034 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:55.764092 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:55.882335 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:55.907487 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:55.936091 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:55.949126 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:55.992036 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:56.051250 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:56.053609 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:56.067889 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:56.074508 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:56.095052 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:56.136807 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:56.228260 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:56.268509 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:56.289795 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:56.301381 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:56.351989 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:56.392185 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:56.432600 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:56.512214 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:56.525312 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:56.540181 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:04:56.578682 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:56.727348 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:04:56.739838 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:56.748080 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:56.909018 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:56.968151 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:57.078366 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:57.171658 15505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 115 similar messages]
W20240502 14:04:57.189503 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:04:57.282585 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:57.351498 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:57.425446 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:57.477557 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:57.506538 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:57.513093 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
I20240502 14:04:57.623647 27798 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:04:57.623805 27798 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:04:57.624253 27798 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:04:57.625391 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:57.625473 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:04:57.625532 15897 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:04:57.625741 27798 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:04:57.651903 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:57.704478 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:57.749598 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:57.763034 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:57.828035 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:57.864966 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:57.983196 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:57.991001 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:58.032591 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:58.120708 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:04:58.193560 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:58.195832 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:58.244765 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:58.261701 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:58.362684 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:58.386576 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:58.417796 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:58.425549 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:58.428293 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:58.441298 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:58.453693 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:04:58.505537 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:58.617498 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:58.628206 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:58.685696 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:58.704711 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:58.722955 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:58.783284 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:58.812024 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:58.823017 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:04:58.860316 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:04:58.865301 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:58.878646 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:58.890789 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:58.928831 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:59.119514 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:04:59.173164 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:04:59.293764 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:59.362272 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:59.413612 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:04:59.418582 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:04:59.531394 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:04:59.781525 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:04:59.806178 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:04:59.878558 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:04:59.988898 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:00.022893 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:00.028666 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:00.056809 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:00.063490 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:00.222252 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 14:05:00.227891 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:00.290967 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:05:00.297928 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:00.311009 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:00.427848 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:00.434260 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:00.467270 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:00.478057 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:00.523008 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:00.695353 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:05:00.714637 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:05:00.837381 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:00.856688 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:00.888540 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:00.965234 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:00.979797 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:01.019449 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:05:01.022925 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:05:01.045018 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:01.072237 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:01.084285 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:01.124645 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:01.203292 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:01.258849 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:05:01.321163 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:01.391309 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:01.430033 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:01.472529 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:01.473848 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:01.482635 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:05:01.505990 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:05:01.511052 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:05:01.530464 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:01.645218 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:01.688024 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:01.717172 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:01.743696 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:05:01.784046 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:01.867551 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:01.988001 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:02.117466 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:02.216313 15505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 14:05:02.383460 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:02.387413 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:02.457114 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:02.457144 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:02.483979 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:02.552517 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:02.571615 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:02.616631 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:02.650523 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:02.746713 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:02.793197 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:02.913828 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:02.925778 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:02.938593 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:02.962157 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:03.025046 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:03.113119 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:03.187989 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:05:03.188573 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:03.197841 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:03.256510 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:03.269385 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:03.305822 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:03.415910 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:03.437779 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:03.512312 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:03.546991 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:03.609064 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:03.646147 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:03.690232 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:03.698623 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:03.701323 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:03.716069 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:03.749378 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:03.768289 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:03.812148 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:03.814570 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:03.848925 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:03.868296 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:03.947155 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:03.967562 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:04.050166 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:04.051687 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:04.076586 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:04.188782 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:04.383014 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:04.391268 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:04.422938 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:04.438082 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:04.776922 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:04.831705 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:04.891005 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:04.980100 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:04.984570 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:05.041886 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:05.128531 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:05.137507 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:05.178746 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:05.197093 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:05.242662 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:05:05.315799 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:05.386330 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:05.386492 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:05.468139 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:05.484259 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:05.579252 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:05.605526 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:05.656039 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:05.676375 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:05.748015 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:05.800467 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:05.907322 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:05.915374 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:05.963024 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:06.012218 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:06.020120 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:06.120262 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:06.123543 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:06.127216 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:06.145956 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:06.150374 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:06.159425 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:06.164249 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:06.195569 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:06.250317 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:06.275177 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:06.381640 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:06.390122 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:06.392889 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:06.478549 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:06.528175 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:06.736119 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:06.747948 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:06.764303 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:06.828284 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:05:06.844041 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:06.877192 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:07.029963 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:07.080953 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:07.126685 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:07.207453 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:07.218547 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:05:07.293980 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:07.397534 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:07.569527 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:07.582690 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:07.608398 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:07.624466 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:07.678715 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:07.813470 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:07.877177 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:07.892833 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:07.965237 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:07.966753 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:07.991326 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:08.028990 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:08.094396 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:08.149127 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:08.167466 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:08.197036 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 14:05:08.298087 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:08.303586 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:08.347324 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:08.365777 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:08.374053 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:08.503135 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:08.532608 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:08.591121 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:08.621348 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:08.625540 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:08.629263 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:08.680550 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:08.755316 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:08.776471 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:08.796856 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:08.796921 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:08.848245 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:08.880570 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:08.890204 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:08.948995 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:09.085861 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:09.093834 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:09.105690 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:09.162487 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:09.320447 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:09.445003 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:09.500772 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:09.538067 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:09.618995 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:09.627214 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:09.787549 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:09.881302 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:09.913372 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:09.922988 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:09.962348 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:10.000509 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:10.175174 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:10.205104 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:10.301906 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:10.365703 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:05:10.372906 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:10.378711 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:10.396593 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:10.467223 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:10.506175 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:10.518275 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:10.596508 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:10.673074 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:10.740818 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:10.758193 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:10.843590 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:10.851235 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:10.862875 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:10.978418 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:11.001631 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:11.010686 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:11.046686 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:11.060691 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:11.073249 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:11.113576 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:11.118886 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:11.209849 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:11.210652 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:11.279635 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:11.285593 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:11.296515 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:11.432863 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:11.502449 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:11.503193 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:11.516403 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:11.520653 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:11.527478 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:11.672896 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:11.848213 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:11.850891 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:11.852448 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:05:11.875233 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:11.985826 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:12.056689 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:12.228225 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:12.274864 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240502 14:05:12.276468 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:12.347446 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:12.389148 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:12.476780 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:12.547317 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:12.660765 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:12.789355 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:12.837922 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:12.839301 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:12.854811 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:12.895872 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:12.990046 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:12.999761 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:13.088317 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:13.093649 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:13.113422 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:13.120182 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:13.186919 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:13.211781 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 14:05:13.243301 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:13.308648 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:13.368227 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:13.373171 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:13.416180 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:13.448079 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:13.452594 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:13.487597 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:13.496119 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:13.497776 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:13.520570 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:13.542122 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:13.680183 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:13.706326 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:13.740164 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:13.878314 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:13.879251 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:13.931733 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:14.017745 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:14.025988 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:14.032099 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:14.083588 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:14.161489 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:14.218173 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:14.234952 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:14.316104 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:14.494565 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:14.579327 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:14.579327 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:14.595647 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:14.632239 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:14.997704 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:15.023950 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:15.058120 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:15.069645 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:15.076489 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:15.129175 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:15.150382 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:15.198626 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:15.333680 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:15.362645 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:15.395033 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:15.434545 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:15.475246 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:15.519840 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:05:15.520503 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:15.565840 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:15.605127 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:15.606196 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:15.617530 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:15.635167 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:15.754510 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:15.764799 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:15.852512 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:15.855922 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:15.884110 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:15.893402 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:15.915897 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:15.965974 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:15.972707 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:16.070070 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:16.086509 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:16.111621 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:16.303819 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:16.369827 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:16.442255 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:16.451869 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:16.482039 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:16.482039 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:16.492928 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:16.497789 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:16.514367 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:05:16.620712 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:16.622776 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:16.717078 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:16.734299 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:16.870554 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:05:16.879364 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:16.975490 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:17.064528 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:17.130661 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:17.132416 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:17.279186 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 123 similar messages]
I20240502 14:05:17.396955 28035 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:17.397127 28035 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:05:17.397593 28035 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:05:17.398633 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:17.398710 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:05:17.398792 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:05:17.398922 28035 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:05:17.416244 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:17.445909 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:17.577471 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:17.589197 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:17.614491 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:17.631440 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:17.635246 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:17.647147 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:17.669862 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:17.809831 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:17.848954 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:17.946846 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:17.948273 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:17.991765 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:18.022858 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:18.038856 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:18.050325 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:18.071573 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:18.114063 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:18.167695 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:18.197417 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:18.201620 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:18.254034 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 14:05:18.271698 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:18.333737 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:18.334389 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:18.386127 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:18.447633 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:18.456804 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:18.518278 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:18.609059 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:18.807185 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:18.854564 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:18.894639 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:18.932374 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:18.949846 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:19.039009 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:19.053205 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:19.061098 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:19.110309 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:05:19.124047 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:19.191381 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:19.244580 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:19.245142 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:19.418953 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:19.443197 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:19.456182 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:19.741902 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:19.824823 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:19.860169 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:19.876062 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:19.907753 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:20.047905 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:20.057060 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:20.169370 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:20.173767 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:20.185066 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:20.196727 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:20.200703 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:20.291785 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:20.362951 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:20.406649 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:20.449383 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:20.496925 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:20.521874 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:20.563097 15898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 14:05:20.563788 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:20.589843 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:20.643033 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:20.671308 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:20.718901 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:20.722259 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:20.834009 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:20.878945 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:20.905500 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:21.014556 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:21.034904 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:21.035395 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:21.068915 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:21.073345 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:21.117345 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:21.214468 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:21.357508 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:21.382453 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:21.449103 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:21.453871 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:21.478889 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:21.488816 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:21.541196 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:21.590364 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:21.632910 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:21.663686 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:05:21.737922 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:21.841629 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:21.855769 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:21.905992 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240502 14:05:21.929352 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:22.101987 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:22.281253 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:05:22.281982 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:22.328039 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:22.418808 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:22.451855 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:22.471979 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:22.487308 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:22.580219 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:22.580401 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:22.594671 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:22.672763 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:22.681413 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:22.711958 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:22.772330 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:22.775558 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:22.823197 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:22.874684 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:22.999670 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:23.082095 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:23.149857 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:23.165720 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:23.215768 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:23.265507 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:23.280974 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:23.323297 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:23.326196 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 14:05:23.326916 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:23.336548 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:23.339988 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:23.457487 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:23.501045 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:23.506413 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:23.558264 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:23.725276 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:23.744004 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:23.775745 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:23.776353 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:23.849912 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:23.999049 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:24.094688 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:24.100243 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:24.101456 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:24.112457 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:24.145566 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:24.237834 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:24.254084 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:05:24.329582 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:24.507898 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:24.550170 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:24.614445 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:24.615670 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:24.706055 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:24.723623 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:24.944907 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:24.989135 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:25.018101 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:25.031543 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:25.040844 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:25.080564 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:25.108021 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:25.109149 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:25.134929 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:25.139753 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:25.185771 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:25.338759 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:25.413352 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:25.420426 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:25.544997 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:25.581661 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:05:25.626817 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:25.661830 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:25.674510 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:25.731720 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:25.764875 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:25.782229 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:25.857478 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:25.887575 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:25.912560 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:25.966360 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:26.007700 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:26.040571 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:26.105530 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:26.112107 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:26.121922 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:26.163035 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:26.295348 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:26.373903 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:26.449311 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:26.515112 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:26.519145 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:26.534440 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:26.582026 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:26.601294 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:26.754482 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:26.761890 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:26.884615 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:26.886008 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:26.888154 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:05:26.933754 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 120 similar messages]
I20240502 14:05:26.969693 28035 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:26.969833 28035 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:05:26.970311 28035 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:05:26.971289 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:26.971383 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:05:26.971470 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:05:26.971621 28035 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:05:26.979543 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:27.011183 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:27.055133 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:27.183979 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:27.187321 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:27.190208 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:27.343556 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:05:27.381220 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:27.401379 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:27.416064 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:27.541872 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:27.557384 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:27.584447 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:27.591459 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:27.603358 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:27.813432 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:27.859520 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:27.984228 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:27.988158 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:28.004549 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:28.036274 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:28.090435 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:28.094699 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:28.258495 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:28.261857 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:28.285517 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:28.323186 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:28.367018 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:28.375380 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:05:28.376024 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:28.381966 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:28.403352 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:28.412320 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:28.420601 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:28.474114 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:28.510891 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:28.590819 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:28.757776 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:28.847795 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:28.895526 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:28.901154 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:28.905764 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:29.047494 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:29.115258 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:29.128065 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:29.161049 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:29.316449 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:05:29.376679 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:29.401234 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:29.418051 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:29.496026 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:29.503820 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:29.563201 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:29.585299 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:29.650068 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:29.847802 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:29.873464 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:29.947463 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:29.973346 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:29.985030 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:30.004544 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:05:30.035696 28747 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:30.035858 28747 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:05:30.036273 28747 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:05:30.037376 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:30.037957 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:05:30.038033 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:05:30.038167 28747 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:05:30.078006 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:30.214386 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:30.237164 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:30.259297 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:30.288604 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:30.374819 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:30.384234 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:30.436547 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:30.486727 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:30.535516 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:30.595765 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:30.607069 15898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 14:05:30.607717 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:30.698246 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:30.727066 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:30.764328 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:30.788478 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:30.813832 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:30.901300 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:30.905329 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:30.908061 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:30.982882 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:31.110970 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:31.155092 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:31.173640 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:31.256753 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:31.268774 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:31.407222 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:31.528368 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:31.631458 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:31.645025 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:31.704070 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:31.717398 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:31.729629 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:31.735390 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:31.833724 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:31.928398 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:31.941540 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:05:31.948968 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:31.955727 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:31.987898 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240502 14:05:32.037695 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:32.107628 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:32.136830 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:32.179073 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:32.426342 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:05:32.448055 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:32.449250 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:32.491959 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:32.502477 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:32.613571 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:32.614813 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:32.674369 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:32.784744 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:32.807070 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:32.847728 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:32.868058 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:32.876482 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:32.892501 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:32.973521 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:33.058244 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:33.152542 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:33.213330 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:33.214089 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:33.241479 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:33.269378 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:33.287830 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:33.335687 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:33.366505 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:33.390784 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:05:33.393568 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:33.452728 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:33.501356 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:33.512681 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:33.537338 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:33.559302 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:33.898569 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:33.900046 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:33.903249 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:33.943101 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:33.998112 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:34.020022 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:34.036834 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:34.067631 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:34.187913 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:34.202467 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:34.357606 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:34.368748 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:34.371238 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:34.495143 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:34.512148 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:34.617834 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:34.647653 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:05:34.736937 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:34.744880 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:34.832051 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:34.866201 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:34.897615 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:35.050137 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:35.146543 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:35.179533 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:35.244628 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:35.253806 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:35.263387 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:35.271015 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:35.388470 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:35.478888 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:35.550444 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:35.579555 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:35.589048 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:35.631913 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:35.712935 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:05:35.713567 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:35.776372 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:35.842386 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:35.884984 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:35.916123 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:35.939662 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:36.011868 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:36.030113 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:36.032163 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:36.066990 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:36.091048 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:36.124119 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:36.283923 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:36.378147 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:36.425457 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:36.434710 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:36.455238 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:36.555554 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:36.585219 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:36.686674 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:36.742209 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:36.749771 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:36.753980 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:36.775573 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:36.816085 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:36.861690 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:36.923149 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:36.926821 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:37.004204 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:05:37.033972 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:37.129192 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:37.163408 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:05:37.370543 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:37.371040 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:37.411948 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:37.479012 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240502 14:05:37.494526 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:37.602315 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:05:37.644044 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:37.661309 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:37.695952 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:37.704003 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:37.754326 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:05:37.783881 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:37.840327 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:37.871151 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:37.884856 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:37.920857 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:37.953078 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:38.009883 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:05:38.196981 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:38.232561 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:38.319082 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:38.334798 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:38.351089 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:38.394683 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:38.414361 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:38.509567 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:38.511307 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:38.523247 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:38.527743 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 14:05:38.771293 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:38.786342 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:38.838373 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:38.882232 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:38.942346 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:38.988072 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:39.016670 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:39.076898 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:39.102752 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:39.113276 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:39.166177 15767 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:05:39.170140 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:39.231626 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:39.285351 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:05:39.306912 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:05:39.346247 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:05:39.370043 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:05:39.387849 28991 consensus_queue.cc:579] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Leader has been unable to successfully communicate with peer 2254b3ee3db647598d781e467584f78d for more than 300 seconds (300.039s)
I20240502 14:05:39.393833 15700 consensus_queue.cc:237] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:05:39.395601 15960 raft_consensus.cc:1261] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:05:39.395704 15637 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6d365c10c76416a944ec8fe409e3dc5. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:39.396013 29003 consensus_queue.cc:1035] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:05:39.396173 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:39.397794 28991 raft_consensus.cc:2932] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:05:39.398226 15960 raft_consensus.cc:2932] T b6d365c10c76416a944ec8fe409e3dc5 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:05:39.399463 15413 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b6d365c10c76416a944ec8fe409e3dc5 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:05:39.400409 15428 catalog_manager.cc:5484] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 6 to 7, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:39.541579 28747 consensus_queue.cc:579] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Leader has been unable to successfully communicate with peer 2254b3ee3db647598d781e467584f78d for more than 300 seconds (300.157s)
I20240502 14:05:39.548121 15960 consensus_queue.cc:237] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:05:39.548858 28952 consensus_queue.cc:579] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Leader has been unable to successfully communicate with peer 2254b3ee3db647598d781e467584f78d for more than 300 seconds (300.062s)
I20240502 14:05:39.550160 17515 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240502 14:05:39.551110 15504 consensus_peers.cc:462] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:39.551286 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:39.552253 29017 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 14:05:39.554332 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): Couldn't send request to peer 8484181c4ece483c8b408817fab1a917. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e48dda3e62e14e46b03d79648e3e2773. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:39.554402 29018 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } } }
I20240502 14:05:39.554972 29021 consensus_queue.cc:579] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Leader has been unable to successfully communicate with peer 2254b3ee3db647598d781e467584f78d for more than 300 seconds (300.016s)
I20240502 14:05:39.556094 15567 consensus_queue.cc:237] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } }
I20240502 14:05:39.557169 15413 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e48dda3e62e14e46b03d79648e3e2773 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240502 14:05:39.557916 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:05:39.557976 17515 raft_consensus.cc:2932] T e48dda3e62e14e46b03d79648e3e2773 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } } }
I20240502 14:05:39.558588 15960 raft_consensus.cc:1261] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 14:05:39.558627 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:39.558944 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:39.559348 15426 catalog_manager.cc:5484] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 11 to 12, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:39.559705 28968 consensus_queue.cc:1035] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:05:39.562150 28968 raft_consensus.cc:2932] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:05:39.563601 15960 raft_consensus.cc:2932] T b0cd21dc897b41aca9118be6df48cc0c P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } }
I20240502 14:05:39.563999 15415 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b0cd21dc897b41aca9118be6df48cc0c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:05:39.566255 15428 catalog_manager.cc:5484] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 8 to 9, NON_VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:05:39.568961 15413 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:39.569674 15413 catalog_manager.cc:4573] TS 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b0cd21dc897b41aca9118be6df48cc0c: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
I20240502 14:05:39.589571 29021 consensus_queue.cc:579] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Leader has been unable to successfully communicate with peer 2254b3ee3db647598d781e467584f78d for more than 300 seconds (300.294s)
I20240502 14:05:39.592713 15567 consensus_queue.cc:237] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } }
I20240502 14:05:39.594408 17515 raft_consensus.cc:1261] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 14:05:39.594588 15507 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531): Couldn't send request to peer 82d7b7201cfc431d90baca1ad0a397cf. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:39.594792 29021 consensus_queue.cc:1035] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 14:05:39.594964 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:39.596659 28968 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } } }
I20240502 14:05:39.597100 17515 raft_consensus.cc:2932] T d81f14b7c1d1468196d0fe6d67fe93a7 P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } } }
I20240502 14:05:39.598440 15415 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d81f14b7c1d1468196d0fe6d67fe93a7 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:05:39.599551 15426 catalog_manager.cc:5484] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 9 to 10, NON_VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:05:39.604573 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:05:39.639801 28991 consensus_queue.cc:579] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Leader has been unable to successfully communicate with peer 2254b3ee3db647598d781e467584f78d for more than 300 seconds (300.310s)
W20240502 14:05:39.641216 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:05:39.642657 15700 consensus_queue.cc:237] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } }
W20240502 14:05:39.644482 15638 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531): Couldn't send request to peer 82d7b7201cfc431d90baca1ad0a397cf. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:39.644577 15567 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 14:05:39.644922 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:39.645025 28991 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:05:39.646802 28904 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } } }
I20240502 14:05:39.647248 15567 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } } }
I20240502 14:05:39.648845 15413 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 76f07464d9bd49a9b1b9d58f6a180cf8 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:05:39.649448 15428 catalog_manager.cc:5484] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 8 to 9, NON_VOTER 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:39.712967 28991 consensus_queue.cc:579] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Leader has been unable to successfully communicate with peer 2254b3ee3db647598d781e467584f78d for more than 300 seconds (300.153s)
I20240502 14:05:39.764808 28974 consensus_queue.cc:579] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Leader has been unable to successfully communicate with peer 3c3658c946994e2ba9f41b64aa327d25 for more than 300 seconds (300.414s)
W20240502 14:05:39.766340 15766 consensus_peers.cc:462] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:05:39.770835 15831 consensus_queue.cc:237] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:05:39.772374 17515 raft_consensus.cc:1261] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:05:39.772393 15567 raft_consensus.cc:1261] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:05:39.772742 28974 consensus_queue.cc:1035] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:05:39.772894 29041 consensus_queue.cc:1035] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:05:39.774758 29042 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:05:39.775202 15567 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:05:39.775247 17515 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:05:39.777179 15416 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 28e371c1a999431499e09479e8f5b44b with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:05:39.778081 15428 catalog_manager.cc:5484] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 7 to 8, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:05:39.781806 15831 consensus_queue.cc:237] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
W20240502 14:05:39.783381 15766 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): Couldn't send request to peer 8484181c4ece483c8b408817fab1a917. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28e371c1a999431499e09479e8f5b44b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:39.783654 15569 raft_consensus.cc:1261] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:05:39.784341 29042 consensus_queue.cc:1035] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:05:39.784389 17515 raft_consensus.cc:1261] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:05:39.784853 29042 consensus_queue.cc:1035] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:05:39.785902 29041 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } } }
I20240502 14:05:39.786407 15569 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } } }
I20240502 14:05:39.787284 15416 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 28e371c1a999431499e09479e8f5b44b with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:05:39.788650 15426 catalog_manager.cc:5484] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 8 to 9, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:39.788825 17515 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 8fea283177394bd28d7eacf4fb744d59 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } } }
I20240502 14:05:39.793874 29047 ts_tablet_manager.cc:902] T b6d365c10c76416a944ec8fe409e3dc5 P 8fea283177394bd28d7eacf4fb744d59: Initiating tablet copy from peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
F20240502 14:05:39.794081 29047 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-03dd735113ab4ea6bdb696b11a4efd7f.queue_time_us, tablet-download-pool-03dd735113ab4ea6bdb696b11a4efd7f.queue_time_us) (tablet-download-pool-03dd735113ab4ea6bdb696b11a4efd7f.run_cpu_time_us, tablet-download-pool-03dd735113ab4ea6bdb696b11a4efd7f.run_cpu_time_us) (tablet-download-pool-03dd735113ab4ea6bdb696b11a4efd7f.run_wall_time_us, tablet-download-pool-03dd735113ab4ea6bdb696b11a4efd7f.run_wall_time_us) (tablet-download-pool-1a5e33f337bd47b2a6b11b29a5eacad8.queue_time_us, tablet-download-pool-1a5e33f337bd47b2a6b11b29a5eacad8.queue_time_us) (tablet-download-pool-1a5e33f337bd47b2a6b11b29a5eacad8.run_cpu_time_us, tablet-download-pool-1a5e33f337bd47b2a6b11b29a5eacad8.run_cpu_time_us) (tablet-download-pool-1a5e33f337bd47b2a6b11b29a5eacad8.run_wall_time_us, tablet-download-pool-1a5e33f337bd47b2a6b11b29a5eacad8.run_wall_time_us) (tablet-download-pool-22eef55211ca4f65979022d2f4ae829f.queue_time_us, tablet-download-pool-22eef55211ca4f65979022d2f4ae829f.queue_time_us) (tablet-download-pool-22eef55211ca4f65979022d2f4ae829f.run_cpu_time_us, tablet-download-pool-22eef55211ca4f65979022d2f4ae829f.run_cpu_time_us) (tablet-download-pool-22eef55211ca4f65979022d2f4ae829f.run_wall_time_us, tablet-download-pool-22eef55211ca4f65979022d2f4ae829f.run_wall_time_us) (tablet-download-pool-2886dfbec1e141198b0cf7a435fac498.queue_time_us, tablet-download-pool-2886dfbec1e141198b0cf7a435fac498.queue_time_us) (tablet-download-pool-2886dfbec1e141198b0cf7a435fac498.run_cpu_time_us, tablet-download-pool-2886dfbec1e141198b0cf7a435fac498.run_cpu_time_us) (tablet-download-pool-2886dfbec1e141198b0cf7a435fac498.run_wall_time_us, tablet-download-pool-2886dfbec1e141198b0cf7a435fac498.run_wall_time_us) (tablet-download-pool-28e371c1a999431499e09479e8f5b44b.queue_time_us, tablet-download-pool-28e371c1a999431499e09479e8f5b44b.queue_time_us) (tablet-download-pool-28e371c1a999431499e09479e8f5b44b.run_cpu_time_us, tablet-download-pool-28e371c1a999431499e09479e8f5b44b.run_cpu_time_us) (tablet-download-pool-28e371c1a999431499e09479e8f5b44b.run_wall_time_us, tablet-download-pool-28e371c1a999431499e09479e8f5b44b.run_wall_time_us) (tablet-download-pool-2d92e5885bf94c3e8e0e4dee40017699.queue_time_us, tablet-download-pool-2d92e5885bf94c3e8e0e4dee40017699.queue_time_us) (tablet-download-pool-2d92e5885bf94c3e8e0e4dee40017699.run_cpu_time_us, tablet-download-pool-2d92e5885bf94c3e8e0e4dee40017699.run_cpu_time_us) (tablet-download-pool-2d92e5885bf94c3e8e0e4dee40017699.run_wall_time_us, tablet-download-pool-2d92e5885bf94c3e8e0e4dee40017699.run_wall_time_us) (tablet-download-pool-436c127af87c49e8b7bf9b6571f7dc30.queue_time_us, tablet-download-pool-436c127af87c49e8b7bf9b6571f7dc30.queue_time_us) (tablet-download-pool-436c127af87c49e8b7bf9b6571f7dc30.run_cpu_time_us, tablet-download-pool-436c127af87c49e8b7bf9b6571f7dc30.run_cpu_time_us) (tablet-download-pool-436c127af87c49e8b7bf9b6571f7dc30.run_wall_time_us, tablet-download-pool-436c127af87c49e8b7bf9b6571f7dc30.run_wall_time_us) (tablet-download-pool-619a8ec66ba74ecea689c17b4734253a.queue_time_us, tablet-download-pool-619a8ec66ba74ecea689c17b4734253a.queue_time_us) (tablet-download-pool-619a8ec66ba74ecea689c17b4734253a.run_cpu_time_us, tablet-download-pool-619a8ec66ba74ecea689c17b4734253a.run_cpu_time_us) (tablet-download-pool-619a8ec66ba74ecea689c17b4734253a.run_wall_time_us, tablet-download-pool-619a8ec66ba74ecea689c17b4734253a.run_wall_time_us) (tablet-download-pool-6e76bd0ecd3e4c168e141fb7984af3ec.queue_time_us, tablet-download-pool-6e76bd0ecd3e4c168e141fb7984af3ec.queue_time_us) (tablet-download-pool-6e76bd0ecd3e4c168e141fb7984af3ec.run_cpu_time_us, tablet-download-pool-6e76bd0ecd3e4c168e141fb7984af3ec.run_cpu_time_us) (tablet-download-pool-6e76bd0ecd3e4c168e141fb7984af3ec.run_wall_time_us, tablet-download-pool-6e76bd0ecd3e4c168e141fb7984af3ec.run_wall_time_us) (tablet-download-pool-7c57cab68b764db0917ed98e74757ff0.queue_time_us, tablet-download-pool-7c57cab68b764db0917ed98e74757ff0.queue_time_us) (tablet-download-pool-7c57cab68b764db0917ed98e74757ff0.run_cpu_time_us, tablet-download-pool-7c57cab68b764db0917ed98e74757ff0.run_cpu_time_us) (tablet-download-pool-7c57cab68b764db0917ed98e74757ff0.run_wall_time_us, tablet-download-pool-7c57cab68b764db0917ed98e74757ff0.run_wall_time_us) (tablet-download-pool-a7981c3f46d8483caae82fbf2296e762.queue_time_us, tablet-download-pool-a7981c3f46d8483caae82fbf2296e762.queue_time_us) (tablet-download-pool-a7981c3f46d8483caae82fbf2296e762.run_cpu_time_us, tablet-download-pool-a7981c3f46d8483caae82fbf2296e762.run_cpu_time_us) (tablet-download-pool-a7981c3f46d8483caae82fbf2296e762.run_wall_time_us, tablet-download-pool-a7981c3f46d8483caae82fbf2296e762.run_wall_time_us) (tablet-download-pool-b6d365c10c76416a944ec8fe409e3dc5.queue_time_us, tablet-download-pool-b6d365c10c76416a944ec8fe409e3dc5.queue_time_us) (tablet-download-pool-d81f14b7c1d1468196d0fe6d67fe93a7.queue_time_us, tablet-download-pool-d81f14b7c1d1468196d0fe6d67fe93a7.queue_time_us) (tablet-download-pool-d81f14b7c1d1468196d0fe6d67fe93a7.run_cpu_time_us, tablet-download-pool-d81f14b7c1d1468196d0fe6d67fe93a7.run_cpu_time_us) (tablet-download-pool-d81f14b7c1d1468196d0fe6d67fe93a7.run_wall_time_us, tablet-download-pool-d81f14b7c1d1468196d0fe6d67fe93a7.run_wall_time_us) (tablet-download-pool-dfdcf1124e8a474facc14c45e8963568.queue_time_us, tablet-download-pool-dfdcf1124e8a474facc14c45e8963568.queue_time_us) (tablet-download-pool-dfdcf1124e8a474facc14c45e8963568.run_cpu_time_us, tablet-download-pool-dfdcf1124e8a474facc14c45e8963568.run_cpu_time_us) (tablet-download-pool-dfdcf1124e8a474facc14c45e8963568.run_wall_time_us, tablet-download-pool-dfdcf1124e8a474facc14c45e8963568.run_wall_time_us) (tablet-download-pool-e48dda3e62e14e46b03d79648e3e2773.queue_time_us, tablet-download-pool-e48dda3e62e14e46b03d79648e3e2773.queue_time_us) (tablet-download-pool-e48dda3e62e14e46b03d79648e3e2773.run_cpu_time_us, tablet-download-pool-e48dda3e62e14e46b03d79648e3e2773.run_cpu_time_us) (tablet-download-pool-e48dda3e62e14e46b03d79648e3e2773.run_wall_time_us, tablet-download-pool-e48dda3e62e14e46b03d79648e3e2773.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f0ceb2efdcd google::LogMessage::Fail() at ??:0
@ 0x7f0ceb2f3b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f0ceb2ef7cc google::LogMessage::Flush() at ??:0
@ 0x7f0ceb2f0f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0cec759044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f0cec73df98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f0cec73befc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f0cef2ee77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f0cef2f7a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f0cef37876f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f0cef38c4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f0cef377226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f0cef386c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5597d2ec975c std::function<>::operator()() at ??:0
@ 0x7f0cec740e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0cec7416ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f0cec74303b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5597d2ec975c std::function<>::operator()() at ??:0
@ 0x7f0cec730807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0cecf6e6db start_thread at ??:0
@ 0x7f0cea4a671f clone at ??:0
I20240502 14:05:39.839322 29026 consensus_queue.cc:579] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Leader has been unable to successfully communicate with peer 2254b3ee3db647598d781e467584f78d for more than 300 seconds (300.451s)
I20240502 14:05:39.842736 15960 consensus_queue.cc:237] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:05:39.844406 15569 raft_consensus.cc:1261] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:05:39.844772 29026 consensus_queue.cc:1035] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 14:05:39.846269 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:39.846849 29017 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:05:39.847333 15569 raft_consensus.cc:2932] T 837e8b1235a44131bdaef24b763d1cac P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:05:39.848316 15413 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 837e8b1235a44131bdaef24b763d1cac with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:05:39.849514 15428 catalog_manager.cc:5484] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 9 to 10, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:39.857000 29025 consensus_queue.cc:579] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Leader has been unable to successfully communicate with peer 3c3658c946994e2ba9f41b64aa327d25 for more than 300 seconds (300.198s)
I20240502 14:05:39.859387 15960 consensus_queue.cc:237] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:05:39.860899 15831 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:05:39.861305 29017 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 14:05:39.861534 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): Couldn't send request to peer 8484181c4ece483c8b408817fab1a917. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:39.861770 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:39.863536 29025 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } } }
I20240502 14:05:39.864005 15831 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } } }
I20240502 14:05:39.865262 15413 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9391f16ae2d247ec8a71e5c971cdda71 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:05:39.866725 15426 catalog_manager.cc:5484] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 8 to 9, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:39.890650 28974 consensus_queue.cc:579] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Leader has been unable to successfully communicate with peer 2254b3ee3db647598d781e467584f78d for more than 300 seconds (300.434s)
W20240502 14:05:39.893196 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:05:39.949469 29007 consensus_queue.cc:579] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Leader has been unable to successfully communicate with peer 3c3658c946994e2ba9f41b64aa327d25 for more than 300 seconds (300.364s)
I20240502 14:05:39.952525 15700 consensus_queue.cc:237] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:05:39.954214 15569 raft_consensus.cc:1261] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:05:39.954939 29007 consensus_queue.cc:1035] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 14:05:39.955117 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:39.955334 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:39.955963 29017 consensus_queue.cc:579] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Leader has been unable to successfully communicate with peer f688370715454d6e814a78fcf85ac5c3 for more than 300 seconds (300.048s)
I20240502 14:05:39.957229 28991 raft_consensus.cc:2932] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
W20240502 14:05:39.958761 15898 consensus_peers.cc:462] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:05:39.959609 15960 consensus_queue.cc:237] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:05:39.960000 15428 catalog_manager.cc:5484] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 7 to 8, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:39.961081 15413 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1f80c8590e994a10a9b7c6c174e28173 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:05:39.961114 15831 raft_consensus.cc:1261] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:05:39.961551 29017 consensus_queue.cc:1035] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:05:39.961514 15569 raft_consensus.cc:2932] T 1f80c8590e994a10a9b7c6c174e28173 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: NON_VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:05:39.963348 28747 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:05:39.965060 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1a5e33f337bd47b2a6b11b29a5eacad8 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:05:39.966480 15831 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:05:39.968420 15426 catalog_manager.cc:5484] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 7 to 8, NON_VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:39.971611 15960 consensus_queue.cc:237] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:05:39.973531 15831 raft_consensus.cc:1261] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 14:05:39.974205 15897 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): Couldn't send request to peer 8484181c4ece483c8b408817fab1a917. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a5e33f337bd47b2a6b11b29a5eacad8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:39.974335 29017 consensus_queue.cc:1035] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:05:39.975919 29026 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } } }
I20240502 14:05:39.976352 15831 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } } }
I20240502 14:05:39.977999 15413 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1a5e33f337bd47b2a6b11b29a5eacad8 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:05:39.979353 15428 catalog_manager.cc:5484] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 8 to 9, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:39.991636 29007 consensus_queue.cc:579] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Leader has been unable to successfully communicate with peer 3c3658c946994e2ba9f41b64aa327d25 for more than 300 seconds (300.082s)
I20240502 14:05:39.994482 15700 consensus_queue.cc:237] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } }
W20240502 14:05:39.996387 15638 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531): Couldn't send request to peer 82d7b7201cfc431d90baca1ad0a397cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7981c3f46d8483caae82fbf2296e762. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:39.996635 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:40.013235 28952 consensus_queue.cc:579] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Leader has been unable to successfully communicate with peer 3c3658c946994e2ba9f41b64aa327d25 for more than 300 seconds (300.432s)
I20240502 14:05:40.016301 15569 consensus_queue.cc:237] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:05:40.018174 15831 raft_consensus.cc:1261] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:05:40.018649 28968 consensus_queue.cc:1035] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 14:05:40.018864 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:40.020648 28952 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:05:40.021150 15831 raft_consensus.cc:2932] T 186a946321e445b58b0097c658847ebe P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:05:40.022954 15415 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 186a946321e445b58b0097c658847ebe with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:05:40.023679 15426 catalog_manager.cc:5484] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 8 to 9, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:05:40.030069 15637 connection.cc:535] client connection to 127.8.86.135:46245 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:05:40.030094 15899 connection.cc:535] client connection to 127.8.86.135:46245 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:05:40.030138 15506 connection.cc:535] client connection to 127.8.86.135:46245 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:05:40.030378 15768 connection.cc:535] client connection to 127.8.86.135:46245 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:05:40.031153 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:40.031239 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:40.031229 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:40.031294 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:40.031356 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:40.031685 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:40.031749 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:40.031806 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:40.031855 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:40.031847 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:40.031903 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:40.031912 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:40.086433 29091 ts_tablet_manager.cc:902] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: Initiating tablet copy from peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:05:40.087047 29091 tablet_copy_client.cc:308] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:42261
I20240502 14:05:40.087673 15721 tablet_copy_service.cc:140] P 8484181c4ece483c8b408817fab1a917: Received BeginTabletCopySession request for tablet 76f07464d9bd49a9b1b9d58f6a180cf8 from peer 82d7b7201cfc431d90baca1ad0a397cf ({username='slave'} at 127.8.86.131:52475)
I20240502 14:05:40.087764 15721 tablet_copy_service.cc:161] P 8484181c4ece483c8b408817fab1a917: Beginning new tablet copy session on tablet 76f07464d9bd49a9b1b9d58f6a180cf8 from peer 82d7b7201cfc431d90baca1ad0a397cf at {username='slave'} at 127.8.86.131:52475: session id = 82d7b7201cfc431d90baca1ad0a397cf-76f07464d9bd49a9b1b9d58f6a180cf8
I20240502 14:05:40.089027 15721 tablet_copy_source_session.cc:205] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:05:40.090186 29091 ts_tablet_manager.cc:1875] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:05:40.092345 29091 ts_tablet_manager.cc:1888] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240502 14:05:40.092592 29091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f07464d9bd49a9b1b9d58f6a180cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:40.093695 29091 tablet_copy_client.cc:783] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: tablet copy: Starting download of 5 data blocks...
I20240502 14:05:40.098505 29091 tablet_copy_client.cc:647] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: tablet copy: Starting download of 1 WAL segments...
I20240502 14:05:40.099756 29091 tablet_copy_client.cc:523] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:05:40.101330 29091 tablet_bootstrap.cc:492] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:05:40.110159 29091 tablet_bootstrap.cc:492] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:05:40.110510 29091 tablet_bootstrap.cc:492] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap complete.
I20240502 14:05:40.110711 29091 ts_tablet_manager.cc:1367] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.003s
I20240502 14:05:40.111140 29091 raft_consensus.cc:348] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } }
I20240502 14:05:40.111259 29091 raft_consensus.cc:729] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: LEARNER
I20240502 14:05:40.111420 29091 consensus_queue.cc:260] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } }
I20240502 14:05:40.111709 29091 ts_tablet_manager.cc:1398] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:05:40.112267 15721 tablet_copy_service.cc:342] P 8484181c4ece483c8b408817fab1a917: Request end of tablet copy session 82d7b7201cfc431d90baca1ad0a397cf-76f07464d9bd49a9b1b9d58f6a180cf8 received from {username='slave'} at 127.8.86.131:52475
I20240502 14:05:40.112345 15721 tablet_copy_service.cc:433] P 8484181c4ece483c8b408817fab1a917: ending tablet copy session 82d7b7201cfc431d90baca1ad0a397cf-76f07464d9bd49a9b1b9d58f6a180cf8 on tablet 76f07464d9bd49a9b1b9d58f6a180cf8 with peer 82d7b7201cfc431d90baca1ad0a397cf
I20240502 14:05:40.113328 29097 ts_tablet_manager.cc:902] T e48dda3e62e14e46b03d79648e3e2773 P 8484181c4ece483c8b408817fab1a917: Initiating tablet copy from peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:05:40.113912 29097 tablet_copy_client.cc:308] T e48dda3e62e14e46b03d79648e3e2773 P 8484181c4ece483c8b408817fab1a917: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.132:45719
I20240502 14:05:40.114455 15984 tablet_copy_service.cc:140] P 4d0f8f311e834d81bd54b9f8e7923bcd: Received BeginTabletCopySession request for tablet e48dda3e62e14e46b03d79648e3e2773 from peer 8484181c4ece483c8b408817fab1a917 ({username='slave'} at 127.8.86.130:34301)
I20240502 14:05:40.114565 15984 tablet_copy_service.cc:161] P 4d0f8f311e834d81bd54b9f8e7923bcd: Beginning new tablet copy session on tablet e48dda3e62e14e46b03d79648e3e2773 from peer 8484181c4ece483c8b408817fab1a917 at {username='slave'} at 127.8.86.130:34301: session id = 8484181c4ece483c8b408817fab1a917-e48dda3e62e14e46b03d79648e3e2773
I20240502 14:05:40.115756 15984 tablet_copy_source_session.cc:205] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd: Tablet Copy: opened 5 blocks and 1 log segments
W20240502 14:05:40.116606 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:40.116803 29097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48dda3e62e14e46b03d79648e3e2773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:40.119005 29097 tablet_copy_client.cc:783] T e48dda3e62e14e46b03d79648e3e2773 P 8484181c4ece483c8b408817fab1a917: tablet copy: Starting download of 5 data blocks...
I20240502 14:05:40.123505 29097 tablet_copy_client.cc:647] T e48dda3e62e14e46b03d79648e3e2773 P 8484181c4ece483c8b408817fab1a917: tablet copy: Starting download of 1 WAL segments...
I20240502 14:05:40.124825 29097 tablet_copy_client.cc:523] T e48dda3e62e14e46b03d79648e3e2773 P 8484181c4ece483c8b408817fab1a917: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:05:40.126183 29097 tablet_bootstrap.cc:492] T e48dda3e62e14e46b03d79648e3e2773 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
W20240502 14:05:40.134697 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:40.134737 29097 tablet_bootstrap.cc:492] T e48dda3e62e14e46b03d79648e3e2773 P 8484181c4ece483c8b408817fab1a917: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:05:40.134979 29097 tablet_bootstrap.cc:492] T e48dda3e62e14e46b03d79648e3e2773 P 8484181c4ece483c8b408817fab1a917: Bootstrap complete.
I20240502 14:05:40.135166 29097 ts_tablet_manager.cc:1367] T e48dda3e62e14e46b03d79648e3e2773 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.009s user 0.010s sys 0.000s
I20240502 14:05:40.135552 29097 raft_consensus.cc:348] T e48dda3e62e14e46b03d79648e3e2773 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:05:40.135660 29097 raft_consensus.cc:729] T e48dda3e62e14e46b03d79648e3e2773 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: LEARNER
I20240502 14:05:40.135759 29097 consensus_queue.cc:260] T e48dda3e62e14e46b03d79648e3e2773 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:05:40.136055 29097 ts_tablet_manager.cc:1398] T e48dda3e62e14e46b03d79648e3e2773 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:05:40.136606 15984 tablet_copy_service.cc:342] P 4d0f8f311e834d81bd54b9f8e7923bcd: Request end of tablet copy session 8484181c4ece483c8b408817fab1a917-e48dda3e62e14e46b03d79648e3e2773 received from {username='slave'} at 127.8.86.130:34301
I20240502 14:05:40.136683 15984 tablet_copy_service.cc:433] P 4d0f8f311e834d81bd54b9f8e7923bcd: ending tablet copy session 8484181c4ece483c8b408817fab1a917-e48dda3e62e14e46b03d79648e3e2773 on tablet e48dda3e62e14e46b03d79648e3e2773 with peer 8484181c4ece483c8b408817fab1a917
I20240502 14:05:40.158732 29021 consensus_queue.cc:579] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Leader has been unable to successfully communicate with peer 3c3658c946994e2ba9f41b64aa327d25 for more than 300 seconds (300.420s)
I20240502 14:05:40.161340 15569 consensus_queue.cc:237] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:05:40.163040 15700 raft_consensus.cc:1261] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:05:40.163390 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:40.163453 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:40.163687 29021 consensus_queue.cc:1035] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:05:40.165441 28968 raft_consensus.cc:2932] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:05:40.165930 15700 raft_consensus.cc:2932] T 9870bc797f184a6284ff6ca4f78b3826 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:05:40.166839 15415 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9870bc797f184a6284ff6ca4f78b3826 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:05:40.168249 15428 catalog_manager.cc:5484] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf reported cstate change: config changed from index 6 to 7, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:05:40.168593 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:05:40.169054 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:40.199299 29097 ts_tablet_manager.cc:902] T 28e371c1a999431499e09479e8f5b44b P 8484181c4ece483c8b408817fab1a917: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:05:40.199816 29097 tablet_copy_client.cc:308] T 28e371c1a999431499e09479e8f5b44b P 8484181c4ece483c8b408817fab1a917: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
I20240502 14:05:40.200300 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 28e371c1a999431499e09479e8f5b44b from peer 8484181c4ece483c8b408817fab1a917 ({username='slave'} at 127.8.86.130:49307)
I20240502 14:05:40.200426 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 28e371c1a999431499e09479e8f5b44b from peer 8484181c4ece483c8b408817fab1a917 at {username='slave'} at 127.8.86.130:49307: session id = 8484181c4ece483c8b408817fab1a917-28e371c1a999431499e09479e8f5b44b
I20240502 14:05:40.201700 15853 tablet_copy_source_session.cc:205] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:05:40.202622 29097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e371c1a999431499e09479e8f5b44b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:40.204813 29097 tablet_copy_client.cc:783] T 28e371c1a999431499e09479e8f5b44b P 8484181c4ece483c8b408817fab1a917: tablet copy: Starting download of 5 data blocks...
W20240502 14:05:40.205622 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:05:40.209239 29017 consensus_queue.cc:579] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Leader has been unable to successfully communicate with peer 3c3658c946994e2ba9f41b64aa327d25 for more than 300 seconds (300.359s)
I20240502 14:05:40.209298 29097 tablet_copy_client.cc:647] T 28e371c1a999431499e09479e8f5b44b P 8484181c4ece483c8b408817fab1a917: tablet copy: Starting download of 1 WAL segments...
W20240502 14:05:40.211225 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:40.211414 29097 tablet_copy_client.cc:523] T 28e371c1a999431499e09479e8f5b44b P 8484181c4ece483c8b408817fab1a917: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:05:40.212663 29097 tablet_bootstrap.cc:492] T 28e371c1a999431499e09479e8f5b44b P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:05:40.212939 15960 consensus_queue.cc:237] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:05:40.214391 15831 raft_consensus.cc:1261] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:05:40.214906 29026 consensus_queue.cc:1035] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 14:05:40.215435 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:40.217087 29017 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:05:40.217648 15831 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:05:40.218909 29091 ts_tablet_manager.cc:902] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: Initiating tablet copy from peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237)
I20240502 14:05:40.219406 29091 tablet_copy_client.cc:308] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.129:36237
I20240502 14:05:40.219391 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d92e5885bf94c3e8e0e4dee40017699 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:05:40.219923 15590 tablet_copy_service.cc:140] P 1eb0e3a6c782469fa8110f6bb6a585cf: Received BeginTabletCopySession request for tablet d81f14b7c1d1468196d0fe6d67fe93a7 from peer 82d7b7201cfc431d90baca1ad0a397cf ({username='slave'} at 127.8.86.131:40607)
I20240502 14:05:40.220021 15590 tablet_copy_service.cc:161] P 1eb0e3a6c782469fa8110f6bb6a585cf: Beginning new tablet copy session on tablet d81f14b7c1d1468196d0fe6d67fe93a7 from peer 82d7b7201cfc431d90baca1ad0a397cf at {username='slave'} at 127.8.86.131:40607: session id = 82d7b7201cfc431d90baca1ad0a397cf-d81f14b7c1d1468196d0fe6d67fe93a7
I20240502 14:05:40.220193 15426 catalog_manager.cc:5484] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 7 to 8, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:40.221407 15590 tablet_copy_source_session.cc:205] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:05:40.222265 29091 ts_tablet_manager.cc:1875] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:05:40.223644 15960 consensus_queue.cc:237] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:05:40.224539 29091 ts_tablet_manager.cc:1888] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240502 14:05:40.224745 29091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81f14b7c1d1468196d0fe6d67fe93a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:40.225082 15831 raft_consensus.cc:1261] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:05:40.225505 29026 consensus_queue.cc:1035] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 14:05:40.225916 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:40.226240 29091 tablet_copy_client.cc:783] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: tablet copy: Starting download of 5 data blocks...
W20240502 14:05:40.226276 15897 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261): Couldn't send request to peer 8484181c4ece483c8b408817fab1a917. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:40.227066 29017 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } } }
I20240502 14:05:40.227494 15831 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } } }
I20240502 14:05:40.227900 29097 tablet_bootstrap.cc:492] T 28e371c1a999431499e09479e8f5b44b P 8484181c4ece483c8b408817fab1a917: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:05:40.228183 29097 tablet_bootstrap.cc:492] T 28e371c1a999431499e09479e8f5b44b P 8484181c4ece483c8b408817fab1a917: Bootstrap complete.
I20240502 14:05:40.228348 29097 ts_tablet_manager.cc:1367] T 28e371c1a999431499e09479e8f5b44b P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20240502 14:05:40.228785 29097 raft_consensus.cc:348] T 28e371c1a999431499e09479e8f5b44b P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:05:40.228885 29097 raft_consensus.cc:729] T 28e371c1a999431499e09479e8f5b44b P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: LEARNER
I20240502 14:05:40.228873 15413 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d92e5885bf94c3e8e0e4dee40017699 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:05:40.229542 29097 consensus_queue.cc:260] T 28e371c1a999431499e09479e8f5b44b P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:05:40.229708 15426 catalog_manager.cc:5484] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 8 to 9, NON_VOTER 8484181c4ece483c8b408817fab1a917 (127.8.86.130) added. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:40.229995 29097 ts_tablet_manager.cc:1398] T 28e371c1a999431499e09479e8f5b44b P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:05:40.230584 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session 8484181c4ece483c8b408817fab1a917-28e371c1a999431499e09479e8f5b44b received from {username='slave'} at 127.8.86.130:49307
I20240502 14:05:40.230667 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session 8484181c4ece483c8b408817fab1a917-28e371c1a999431499e09479e8f5b44b on tablet 28e371c1a999431499e09479e8f5b44b with peer 8484181c4ece483c8b408817fab1a917
I20240502 14:05:40.236407 29097 ts_tablet_manager.cc:902] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: Initiating tablet copy from peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:05:40.236847 29097 tablet_copy_client.cc:308] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.132:45719
I20240502 14:05:40.237498 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling UndoDeltaBlockGCOp(76f07464d9bd49a9b1b9d58f6a180cf8): 4925 bytes on disk
I20240502 14:05:40.237802 15984 tablet_copy_service.cc:140] P 4d0f8f311e834d81bd54b9f8e7923bcd: Received BeginTabletCopySession request for tablet 9391f16ae2d247ec8a71e5c971cdda71 from peer 8484181c4ece483c8b408817fab1a917 ({username='slave'} at 127.8.86.130:34301)
I20240502 14:05:40.237916 15984 tablet_copy_service.cc:161] P 4d0f8f311e834d81bd54b9f8e7923bcd: Beginning new tablet copy session on tablet 9391f16ae2d247ec8a71e5c971cdda71 from peer 8484181c4ece483c8b408817fab1a917 at {username='slave'} at 127.8.86.130:34301: session id = 8484181c4ece483c8b408817fab1a917-9391f16ae2d247ec8a71e5c971cdda71
I20240502 14:05:40.237895 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: UndoDeltaBlockGCOp(76f07464d9bd49a9b1b9d58f6a180cf8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240502 14:05:40.238507 29091 tablet_copy_client.cc:647] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: tablet copy: Starting download of 1 WAL segments...
I20240502 14:05:40.238993 15984 tablet_copy_source_session.cc:205] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:05:40.239694 29097 ts_tablet_manager.cc:1875] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:05:40.239774 29091 tablet_copy_client.cc:523] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:05:40.241072 29091 tablet_bootstrap.cc:492] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:05:40.241509 29097 ts_tablet_manager.cc:1888] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240502 14:05:40.241667 29097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9391f16ae2d247ec8a71e5c971cdda71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:40.242522 29097 tablet_copy_client.cc:783] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: tablet copy: Starting download of 5 data blocks...
I20240502 14:05:40.247021 29097 tablet_copy_client.cc:647] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: tablet copy: Starting download of 1 WAL segments...
I20240502 14:05:40.248162 29097 tablet_copy_client.cc:523] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:05:40.249169 29097 tablet_bootstrap.cc:492] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:05:40.250650 29091 tablet_bootstrap.cc:492] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:05:40.250914 29091 tablet_bootstrap.cc:492] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap complete.
I20240502 14:05:40.251073 29091 ts_tablet_manager.cc:1367] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20240502 14:05:40.251487 29091 raft_consensus.cc:348] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } }
I20240502 14:05:40.251610 29091 raft_consensus.cc:729] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: LEARNER
I20240502 14:05:40.251708 29091 consensus_queue.cc:260] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } }
I20240502 14:05:40.252055 29091 ts_tablet_manager.cc:1398] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 14:05:40.252657 15590 tablet_copy_service.cc:342] P 1eb0e3a6c782469fa8110f6bb6a585cf: Request end of tablet copy session 82d7b7201cfc431d90baca1ad0a397cf-d81f14b7c1d1468196d0fe6d67fe93a7 received from {username='slave'} at 127.8.86.131:40607
I20240502 14:05:40.252737 15590 tablet_copy_service.cc:433] P 1eb0e3a6c782469fa8110f6bb6a585cf: ending tablet copy session 82d7b7201cfc431d90baca1ad0a397cf-d81f14b7c1d1468196d0fe6d67fe93a7 on tablet d81f14b7c1d1468196d0fe6d67fe93a7 with peer 82d7b7201cfc431d90baca1ad0a397cf
I20240502 14:05:40.257297 29097 tablet_bootstrap.cc:492] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:05:40.257525 29097 tablet_bootstrap.cc:492] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: Bootstrap complete.
I20240502 14:05:40.257678 29097 ts_tablet_manager.cc:1367] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20240502 14:05:40.258031 29097 raft_consensus.cc:348] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:05:40.258158 29097 raft_consensus.cc:729] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: LEARNER
I20240502 14:05:40.258281 29097 consensus_queue.cc:260] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:05:40.258586 29097 ts_tablet_manager.cc:1398] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 14:05:40.259222 15984 tablet_copy_service.cc:342] P 4d0f8f311e834d81bd54b9f8e7923bcd: Request end of tablet copy session 8484181c4ece483c8b408817fab1a917-9391f16ae2d247ec8a71e5c971cdda71 received from {username='slave'} at 127.8.86.130:34301
I20240502 14:05:40.259320 15984 tablet_copy_service.cc:433] P 4d0f8f311e834d81bd54b9f8e7923bcd: ending tablet copy session 8484181c4ece483c8b408817fab1a917-9391f16ae2d247ec8a71e5c971cdda71 on tablet 9391f16ae2d247ec8a71e5c971cdda71 with peer 8484181c4ece483c8b408817fab1a917
W20240502 14:05:40.299295 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:40.302796 29042 consensus_queue.cc:579] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Leader has been unable to successfully communicate with peer 3c3658c946994e2ba9f41b64aa327d25 for more than 300 seconds (300.464s)
I20240502 14:05:40.305666 15831 consensus_queue.cc:237] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: true } }
W20240502 14:05:40.307694 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237): Couldn't send request to peer 1eb0e3a6c782469fa8110f6bb6a585cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03dd735113ab4ea6bdb696b11a4efd7f. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:40.307699 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:40.308279 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:40.323166 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling UndoDeltaBlockGCOp(9391f16ae2d247ec8a71e5c971cdda71): 4925 bytes on disk
I20240502 14:05:40.323666 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: UndoDeltaBlockGCOp(9391f16ae2d247ec8a71e5c971cdda71) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240502 14:05:40.324364 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling UndoDeltaBlockGCOp(28e371c1a999431499e09479e8f5b44b): 4840 bytes on disk
I20240502 14:05:40.324719 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: UndoDeltaBlockGCOp(28e371c1a999431499e09479e8f5b44b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
I20240502 14:05:40.325341 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling UndoDeltaBlockGCOp(e48dda3e62e14e46b03d79648e3e2773): 4586 bytes on disk
I20240502 14:05:40.325656 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: UndoDeltaBlockGCOp(e48dda3e62e14e46b03d79648e3e2773) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
I20240502 14:05:40.350369 28904 consensus_queue.cc:579] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Leader has been unable to successfully communicate with peer f688370715454d6e814a78fcf85ac5c3 for more than 300 seconds (300.283s)
W20240502 14:05:40.351888 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:05:40.353147 15700 consensus_queue.cc:237] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:05:40.354811 15569 raft_consensus.cc:1261] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:05:40.355047 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:40.355103 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:40.355260 29003 consensus_queue.cc:1035] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:05:40.357197 29007 raft_consensus.cc:2932] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:05:40.358525 15413 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b07f4db6f82e4a56accdbf1bbbc5f965 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:05:40.358495 15569 raft_consensus.cc:2932] T b07f4db6f82e4a56accdbf1bbbc5f965 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:05:40.359637 15424 catalog_manager.cc:5484] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 6 to 7, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:05:40.380760 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 14:05:40.382755 29091 ts_tablet_manager.cc:902] T a7981c3f46d8483caae82fbf2296e762 P 82d7b7201cfc431d90baca1ad0a397cf: Initiating tablet copy from peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:05:40.383268 29091 tablet_copy_client.cc:308] T a7981c3f46d8483caae82fbf2296e762 P 82d7b7201cfc431d90baca1ad0a397cf: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:42261
I20240502 14:05:40.383739 15721 tablet_copy_service.cc:140] P 8484181c4ece483c8b408817fab1a917: Received BeginTabletCopySession request for tablet a7981c3f46d8483caae82fbf2296e762 from peer 82d7b7201cfc431d90baca1ad0a397cf ({username='slave'} at 127.8.86.131:52475)
I20240502 14:05:40.383846 15721 tablet_copy_service.cc:161] P 8484181c4ece483c8b408817fab1a917: Beginning new tablet copy session on tablet a7981c3f46d8483caae82fbf2296e762 from peer 82d7b7201cfc431d90baca1ad0a397cf at {username='slave'} at 127.8.86.131:52475: session id = 82d7b7201cfc431d90baca1ad0a397cf-a7981c3f46d8483caae82fbf2296e762
I20240502 14:05:40.385008 15721 tablet_copy_source_session.cc:205] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:05:40.385934 29091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7981c3f46d8483caae82fbf2296e762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:40.388034 29091 tablet_copy_client.cc:783] T a7981c3f46d8483caae82fbf2296e762 P 82d7b7201cfc431d90baca1ad0a397cf: tablet copy: Starting download of 5 data blocks...
I20240502 14:05:40.391923 29091 tablet_copy_client.cc:647] T a7981c3f46d8483caae82fbf2296e762 P 82d7b7201cfc431d90baca1ad0a397cf: tablet copy: Starting download of 1 WAL segments...
I20240502 14:05:40.393101 29091 tablet_copy_client.cc:523] T a7981c3f46d8483caae82fbf2296e762 P 82d7b7201cfc431d90baca1ad0a397cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:05:40.394266 29091 tablet_bootstrap.cc:492] T a7981c3f46d8483caae82fbf2296e762 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap starting.
I20240502 14:05:40.400331 29003 consensus_queue.cc:579] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Leader has been unable to successfully communicate with peer f688370715454d6e814a78fcf85ac5c3 for more than 300 seconds (300.387s)
I20240502 14:05:40.403645 29091 tablet_bootstrap.cc:492] T a7981c3f46d8483caae82fbf2296e762 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:05:40.403952 29091 tablet_bootstrap.cc:492] T a7981c3f46d8483caae82fbf2296e762 P 82d7b7201cfc431d90baca1ad0a397cf: Bootstrap complete.
I20240502 14:05:40.404124 29091 ts_tablet_manager.cc:1367] T a7981c3f46d8483caae82fbf2296e762 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240502 14:05:40.404529 29091 raft_consensus.cc:348] T a7981c3f46d8483caae82fbf2296e762 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }
I20240502 14:05:40.404803 29091 raft_consensus.cc:729] T a7981c3f46d8483caae82fbf2296e762 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 82d7b7201cfc431d90baca1ad0a397cf, State: Initialized, Role: LEARNER
I20240502 14:05:40.404908 29091 consensus_queue.cc:260] T a7981c3f46d8483caae82fbf2296e762 P 82d7b7201cfc431d90baca1ad0a397cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } }
I20240502 14:05:40.405225 29091 ts_tablet_manager.cc:1398] T a7981c3f46d8483caae82fbf2296e762 P 82d7b7201cfc431d90baca1ad0a397cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:05:40.405771 15721 tablet_copy_service.cc:342] P 8484181c4ece483c8b408817fab1a917: Request end of tablet copy session 82d7b7201cfc431d90baca1ad0a397cf-a7981c3f46d8483caae82fbf2296e762 received from {username='slave'} at 127.8.86.131:52475
I20240502 14:05:40.405848 15721 tablet_copy_service.cc:433] P 8484181c4ece483c8b408817fab1a917: ending tablet copy session 82d7b7201cfc431d90baca1ad0a397cf-a7981c3f46d8483caae82fbf2296e762 on tablet a7981c3f46d8483caae82fbf2296e762 with peer 82d7b7201cfc431d90baca1ad0a397cf
I20240502 14:05:40.489806 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling UndoDeltaBlockGCOp(a7981c3f46d8483caae82fbf2296e762): 4840 bytes on disk
I20240502 14:05:40.490356 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: UndoDeltaBlockGCOp(a7981c3f46d8483caae82fbf2296e762) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240502 14:05:40.491220 15880 maintenance_manager.cc:419] P 82d7b7201cfc431d90baca1ad0a397cf: Scheduling UndoDeltaBlockGCOp(d81f14b7c1d1468196d0fe6d67fe93a7): 4586 bytes on disk
I20240502 14:05:40.491614 15770 maintenance_manager.cc:643] P 82d7b7201cfc431d90baca1ad0a397cf: UndoDeltaBlockGCOp(d81f14b7c1d1468196d0fe6d67fe93a7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240502 14:05:40.495743 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:05:40.497503 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:05:40.516949 29097 ts_tablet_manager.cc:902] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8484181c4ece483c8b408817fab1a917: Initiating tablet copy from peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:05:40.517434 29097 tablet_copy_client.cc:308] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8484181c4ece483c8b408817fab1a917: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.132:45719
I20240502 14:05:40.517927 15984 tablet_copy_service.cc:140] P 4d0f8f311e834d81bd54b9f8e7923bcd: Received BeginTabletCopySession request for tablet 1a5e33f337bd47b2a6b11b29a5eacad8 from peer 8484181c4ece483c8b408817fab1a917 ({username='slave'} at 127.8.86.130:34301)
I20240502 14:05:40.518044 15984 tablet_copy_service.cc:161] P 4d0f8f311e834d81bd54b9f8e7923bcd: Beginning new tablet copy session on tablet 1a5e33f337bd47b2a6b11b29a5eacad8 from peer 8484181c4ece483c8b408817fab1a917 at {username='slave'} at 127.8.86.130:34301: session id = 8484181c4ece483c8b408817fab1a917-1a5e33f337bd47b2a6b11b29a5eacad8
I20240502 14:05:40.519026 15984 tablet_copy_source_session.cc:205] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:05:40.519937 29097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5e33f337bd47b2a6b11b29a5eacad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:40.521646 29097 tablet_copy_client.cc:783] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8484181c4ece483c8b408817fab1a917: tablet copy: Starting download of 5 data blocks...
I20240502 14:05:40.525564 29097 tablet_copy_client.cc:647] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8484181c4ece483c8b408817fab1a917: tablet copy: Starting download of 1 WAL segments...
I20240502 14:05:40.526861 29097 tablet_copy_client.cc:523] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8484181c4ece483c8b408817fab1a917: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:05:40.528095 29097 tablet_bootstrap.cc:492] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:05:40.535176 29097 tablet_bootstrap.cc:492] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8484181c4ece483c8b408817fab1a917: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:05:40.535413 29097 tablet_bootstrap.cc:492] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8484181c4ece483c8b408817fab1a917: Bootstrap complete.
I20240502 14:05:40.535562 29097 ts_tablet_manager.cc:1367] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20240502 14:05:40.535923 29097 raft_consensus.cc:348] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:05:40.536024 29097 raft_consensus.cc:729] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: LEARNER
I20240502 14:05:40.536134 29097 consensus_queue.cc:260] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:05:40.536442 29097 ts_tablet_manager.cc:1398] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:05:40.537055 15984 tablet_copy_service.cc:342] P 4d0f8f311e834d81bd54b9f8e7923bcd: Request end of tablet copy session 8484181c4ece483c8b408817fab1a917-1a5e33f337bd47b2a6b11b29a5eacad8 received from {username='slave'} at 127.8.86.130:34301
I20240502 14:05:40.537129 15984 tablet_copy_service.cc:433] P 4d0f8f311e834d81bd54b9f8e7923bcd: ending tablet copy session 8484181c4ece483c8b408817fab1a917-1a5e33f337bd47b2a6b11b29a5eacad8 on tablet 1a5e33f337bd47b2a6b11b29a5eacad8 with peer 8484181c4ece483c8b408817fab1a917
I20240502 14:05:40.570542 28974 consensus_queue.cc:579] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Leader has been unable to successfully communicate with peer f688370715454d6e814a78fcf85ac5c3 for more than 300 seconds (300.055s)
I20240502 14:05:40.573079 15831 consensus_queue.cc:237] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:05:40.574460 15960 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:05:40.574852 29042 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 14:05:40.574993 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:40.576452 28974 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:05:40.577349 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling UndoDeltaBlockGCOp(1a5e33f337bd47b2a6b11b29a5eacad8): 4840 bytes on disk
I20240502 14:05:40.577829 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: UndoDeltaBlockGCOp(1a5e33f337bd47b2a6b11b29a5eacad8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240502 14:05:40.578527 15416 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2886dfbec1e141198b0cf7a435fac498 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:05:40.578400 15960 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:05:40.579362 15426 catalog_manager.cc:5484] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 10 to 11, VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 14:05:40.579806 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1a5e33f337bd47b2a6b11b29a5eacad8: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:05:40.582700 15831 consensus_queue.cc:237] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: true } }
I20240502 14:05:40.584074 15960 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240502 14:05:40.584177 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237): Couldn't send request to peer 1eb0e3a6c782469fa8110f6bb6a585cf. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:40.584429 28974 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 14:05:40.584990 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:40.586144 29042 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: true } } }
I20240502 14:05:40.586561 15960 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: true } } }
I20240502 14:05:40.587523 15416 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2886dfbec1e141198b0cf7a435fac498 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:05:40.588795 15426 catalog_manager.cc:5484] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 11 to 12, NON_VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) added. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:40.637296 28952 consensus_queue.cc:579] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Leader has been unable to successfully communicate with peer f688370715454d6e814a78fcf85ac5c3 for more than 300 seconds (300.230s)
W20240502 14:05:40.653558 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:05:40.668988 15700 raft_consensus.cc:1203] T e48dda3e62e14e46b03d79648e3e2773 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240502 14:05:40.693101 15700 raft_consensus.cc:1203] T 28e371c1a999431499e09479e8f5b44b P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 14:05:40.698439 15831 raft_consensus.cc:1203] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240502 14:05:40.734808 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 14:05:40.763384 29157 ts_tablet_manager.cc:902] T 03dd735113ab4ea6bdb696b11a4efd7f P 1eb0e3a6c782469fa8110f6bb6a585cf: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:05:40.763952 29157 tablet_copy_client.cc:308] T 03dd735113ab4ea6bdb696b11a4efd7f P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
I20240502 14:05:40.764514 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 03dd735113ab4ea6bdb696b11a4efd7f from peer 1eb0e3a6c782469fa8110f6bb6a585cf ({username='slave'} at 127.8.86.129:40053)
I20240502 14:05:40.764611 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 03dd735113ab4ea6bdb696b11a4efd7f from peer 1eb0e3a6c782469fa8110f6bb6a585cf at {username='slave'} at 127.8.86.129:40053: session id = 1eb0e3a6c782469fa8110f6bb6a585cf-03dd735113ab4ea6bdb696b11a4efd7f
I20240502 14:05:40.765462 15853 tablet_copy_source_session.cc:205] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:05:40.766645 29157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03dd735113ab4ea6bdb696b11a4efd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:40.768505 29157 tablet_copy_client.cc:783] T 03dd735113ab4ea6bdb696b11a4efd7f P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet copy: Starting download of 5 data blocks...
W20240502 14:05:40.770677 15636 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:05:40.772990 29157 tablet_copy_client.cc:647] T 03dd735113ab4ea6bdb696b11a4efd7f P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet copy: Starting download of 1 WAL segments...
I20240502 14:05:40.774345 29157 tablet_copy_client.cc:523] T 03dd735113ab4ea6bdb696b11a4efd7f P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:05:40.775867 29157 tablet_bootstrap.cc:492] T 03dd735113ab4ea6bdb696b11a4efd7f P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:05:40.784946 29157 tablet_bootstrap.cc:492] T 03dd735113ab4ea6bdb696b11a4efd7f P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:05:40.785274 29157 tablet_bootstrap.cc:492] T 03dd735113ab4ea6bdb696b11a4efd7f P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap complete.
I20240502 14:05:40.785473 29157 ts_tablet_manager.cc:1367] T 03dd735113ab4ea6bdb696b11a4efd7f P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240502 14:05:40.785928 29157 raft_consensus.cc:348] T 03dd735113ab4ea6bdb696b11a4efd7f P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }
I20240502 14:05:40.786136 29157 raft_consensus.cc:729] T 03dd735113ab4ea6bdb696b11a4efd7f P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: LEARNER
I20240502 14:05:40.786238 29157 consensus_queue.cc:260] T 03dd735113ab4ea6bdb696b11a4efd7f P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: true } }
I20240502 14:05:40.786577 29157 ts_tablet_manager.cc:1398] T 03dd735113ab4ea6bdb696b11a4efd7f P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 14:05:40.787236 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session 1eb0e3a6c782469fa8110f6bb6a585cf-03dd735113ab4ea6bdb696b11a4efd7f received from {username='slave'} at 127.8.86.129:40053
I20240502 14:05:40.787320 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session 1eb0e3a6c782469fa8110f6bb6a585cf-03dd735113ab4ea6bdb696b11a4efd7f on tablet 03dd735113ab4ea6bdb696b11a4efd7f with peer 1eb0e3a6c782469fa8110f6bb6a585cf
I20240502 14:05:40.788422 15700 raft_consensus.cc:1203] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 14:05:40.793931 29097 ts_tablet_manager.cc:902] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: Initiating tablet copy from peer 4d0f8f311e834d81bd54b9f8e7923bcd (127.8.86.132:45719)
I20240502 14:05:40.794406 29097 tablet_copy_client.cc:308] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.132:45719
I20240502 14:05:40.794886 15984 tablet_copy_service.cc:140] P 4d0f8f311e834d81bd54b9f8e7923bcd: Received BeginTabletCopySession request for tablet 2d92e5885bf94c3e8e0e4dee40017699 from peer 8484181c4ece483c8b408817fab1a917 ({username='slave'} at 127.8.86.130:34301)
I20240502 14:05:40.794984 15984 tablet_copy_service.cc:161] P 4d0f8f311e834d81bd54b9f8e7923bcd: Beginning new tablet copy session on tablet 2d92e5885bf94c3e8e0e4dee40017699 from peer 8484181c4ece483c8b408817fab1a917 at {username='slave'} at 127.8.86.130:34301: session id = 8484181c4ece483c8b408817fab1a917-2d92e5885bf94c3e8e0e4dee40017699
I20240502 14:05:40.795789 15984 tablet_copy_source_session.cc:205] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:05:40.796468 29097 ts_tablet_manager.cc:1875] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:05:40.798424 29097 ts_tablet_manager.cc:1888] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 14:05:40.798602 29097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d92e5885bf94c3e8e0e4dee40017699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:40.799477 29097 tablet_copy_client.cc:783] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: tablet copy: Starting download of 5 data blocks...
I20240502 14:05:40.804174 29097 tablet_copy_client.cc:647] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: tablet copy: Starting download of 1 WAL segments...
I20240502 14:05:40.805429 29097 tablet_copy_client.cc:523] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:05:40.806707 29097 tablet_bootstrap.cc:492] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: Bootstrap starting.
I20240502 14:05:40.813592 29097 tablet_bootstrap.cc:492] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:05:40.813829 29097 tablet_bootstrap.cc:492] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: Bootstrap complete.
I20240502 14:05:40.813989 29097 ts_tablet_manager.cc:1367] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20240502 14:05:40.814407 29097 raft_consensus.cc:348] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:05:40.814517 29097 raft_consensus.cc:729] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8484181c4ece483c8b408817fab1a917, State: Initialized, Role: LEARNER
I20240502 14:05:40.814625 29097 consensus_queue.cc:260] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: NON_VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: true } }
I20240502 14:05:40.814900 29097 ts_tablet_manager.cc:1398] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 14:05:40.815446 15984 tablet_copy_service.cc:342] P 4d0f8f311e834d81bd54b9f8e7923bcd: Request end of tablet copy session 8484181c4ece483c8b408817fab1a917-2d92e5885bf94c3e8e0e4dee40017699 received from {username='slave'} at 127.8.86.130:34301
I20240502 14:05:40.815526 15984 tablet_copy_service.cc:433] P 4d0f8f311e834d81bd54b9f8e7923bcd: ending tablet copy session 8484181c4ece483c8b408817fab1a917-2d92e5885bf94c3e8e0e4dee40017699 on tablet 2d92e5885bf94c3e8e0e4dee40017699 with peer 8484181c4ece483c8b408817fab1a917
I20240502 14:05:40.829762 15749 maintenance_manager.cc:419] P 8484181c4ece483c8b408817fab1a917: Scheduling UndoDeltaBlockGCOp(2d92e5885bf94c3e8e0e4dee40017699): 4840 bytes on disk
I20240502 14:05:40.830219 15639 maintenance_manager.cc:643] P 8484181c4ece483c8b408817fab1a917: UndoDeltaBlockGCOp(2d92e5885bf94c3e8e0e4dee40017699) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
I20240502 14:05:40.830426 15831 raft_consensus.cc:1203] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240502 14:05:40.840215 29041 consensus_queue.cc:579] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Leader has been unable to successfully communicate with peer f688370715454d6e814a78fcf85ac5c3 for more than 300 seconds (300.419s)
W20240502 14:05:40.883805 15636 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:05:40.926002 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling UndoDeltaBlockGCOp(03dd735113ab4ea6bdb696b11a4efd7f): 4586 bytes on disk
I20240502 14:05:40.926561 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: UndoDeltaBlockGCOp(03dd735113ab4ea6bdb696b11a4efd7f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240502 14:05:40.938675 29003 consensus_queue.cc:579] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Leader has been unable to successfully communicate with peer 3c3658c946994e2ba9f41b64aa327d25 for more than 300 seconds (300.178s)
I20240502 14:05:40.941260 15700 consensus_queue.cc:237] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:05:40.942845 15569 raft_consensus.cc:1261] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 2 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
W20240502 14:05:40.943215 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:40.943289 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:40.943308 28904 consensus_queue.cc:1035] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:05:40.944972 29003 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:05:40.945498 15569 raft_consensus.cc:2932] T 436c127af87c49e8b7bf9b6571f7dc30 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } } }
I20240502 14:05:40.946749 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 436c127af87c49e8b7bf9b6571f7dc30 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:05:40.947824 15424 catalog_manager.cc:5484] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 11 to 13, VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New cstate: current_term: 2 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:40.968858 15831 raft_consensus.cc:1203] T a7981c3f46d8483caae82fbf2296e762 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.13->[1.14-1.14] Dedup: 1.14->[]
W20240502 14:05:40.987442 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:05:41.046774 29026 raft_consensus.cc:1050] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd: attempting to promote NON_VOTER 8484181c4ece483c8b408817fab1a917 to VOTER
I20240502 14:05:41.047199 29026 consensus_queue.cc:237] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } }
I20240502 14:05:41.048565 15700 raft_consensus.cc:1261] T e48dda3e62e14e46b03d79648e3e2773 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 14:05:41.049033 29178 consensus_queue.cc:1035] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240502 14:05:41.049237 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:41.049458 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:41.050714 28968 consensus_queue.cc:579] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Leader has been unable to successfully communicate with peer f688370715454d6e814a78fcf85ac5c3 for more than 300 seconds (300.100s)
I20240502 14:05:41.103159 15700 raft_consensus.cc:1203] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 14:05:41.112600 28968 consensus_queue.cc:579] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Leader has been unable to successfully communicate with peer 3c3658c946994e2ba9f41b64aa327d25 for more than 300 seconds (300.162s)
I20240502 14:05:41.134456 29157 ts_tablet_manager.cc:902] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: Initiating tablet copy from peer 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131:41531)
I20240502 14:05:41.135067 29157 tablet_copy_client.cc:308] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.131:41531
I20240502 14:05:41.135555 15853 tablet_copy_service.cc:140] P 82d7b7201cfc431d90baca1ad0a397cf: Received BeginTabletCopySession request for tablet 2886dfbec1e141198b0cf7a435fac498 from peer 1eb0e3a6c782469fa8110f6bb6a585cf ({username='slave'} at 127.8.86.129:40053)
I20240502 14:05:41.135663 15853 tablet_copy_service.cc:161] P 82d7b7201cfc431d90baca1ad0a397cf: Beginning new tablet copy session on tablet 2886dfbec1e141198b0cf7a435fac498 from peer 1eb0e3a6c782469fa8110f6bb6a585cf at {username='slave'} at 127.8.86.129:40053: session id = 1eb0e3a6c782469fa8110f6bb6a585cf-2886dfbec1e141198b0cf7a435fac498
I20240502 14:05:41.136512 15853 tablet_copy_source_session.cc:205] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:05:41.137253 29157 ts_tablet_manager.cc:1875] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:05:41.139281 29157 ts_tablet_manager.cc:1888] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240502 14:05:41.139465 29157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2886dfbec1e141198b0cf7a435fac498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:41.140625 29157 tablet_copy_client.cc:783] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet copy: Starting download of 5 data blocks...
I20240502 14:05:41.144867 29157 tablet_copy_client.cc:647] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet copy: Starting download of 1 WAL segments...
I20240502 14:05:41.146051 29157 tablet_copy_client.cc:523] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:05:41.147230 29157 tablet_bootstrap.cc:492] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:05:41.155323 29157 tablet_bootstrap.cc:492] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:05:41.155645 29157 tablet_bootstrap.cc:492] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap complete.
I20240502 14:05:41.155827 29157 ts_tablet_manager.cc:1367] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240502 14:05:41.156283 29157 raft_consensus.cc:348] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: true } }
I20240502 14:05:41.156435 29157 raft_consensus.cc:729] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: LEARNER
I20240502 14:05:41.156562 29157 consensus_queue.cc:260] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: true } }
I20240502 14:05:41.156847 29157 ts_tablet_manager.cc:1398] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:05:41.157348 15853 tablet_copy_service.cc:342] P 82d7b7201cfc431d90baca1ad0a397cf: Request end of tablet copy session 1eb0e3a6c782469fa8110f6bb6a585cf-2886dfbec1e141198b0cf7a435fac498 received from {username='slave'} at 127.8.86.129:40053
I20240502 14:05:41.157433 15853 tablet_copy_service.cc:433] P 82d7b7201cfc431d90baca1ad0a397cf: ending tablet copy session 1eb0e3a6c782469fa8110f6bb6a585cf-2886dfbec1e141198b0cf7a435fac498 on tablet 2886dfbec1e141198b0cf7a435fac498 with peer 1eb0e3a6c782469fa8110f6bb6a585cf
I20240502 14:05:41.175082 29176 raft_consensus.cc:1050] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917: attempting to promote NON_VOTER 82d7b7201cfc431d90baca1ad0a397cf to VOTER
I20240502 14:05:41.175596 29176 consensus_queue.cc:237] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: false } }
I20240502 14:05:41.176970 15569 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:05:41.177227 15831 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEARNER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:05:41.177534 28991 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:05:41.177814 28904 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 14:05:41.178329 15635 consensus_peers.cc:485] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:41.178774 29178 raft_consensus.cc:1050] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd: attempting to promote NON_VOTER 8484181c4ece483c8b408817fab1a917 to VOTER
I20240502 14:05:41.178773 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling UndoDeltaBlockGCOp(2886dfbec1e141198b0cf7a435fac498): 4924 bytes on disk
I20240502 14:05:41.179195 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: UndoDeltaBlockGCOp(2886dfbec1e141198b0cf7a435fac498) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240502 14:05:41.179222 29178 consensus_queue.cc:237] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } }
I20240502 14:05:41.180227 28991 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: false } } }
I20240502 14:05:41.180714 15700 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:05:41.180822 15831 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: false } } }
I20240502 14:05:41.181401 29017 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:05:41.182504 15569 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: false } } }
I20240502 14:05:41.183004 15832 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 14:05:41.183194 15897 consensus_peers.cc:485] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:41.183751 15424 catalog_manager.cc:5484] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 9 to 10, 82d7b7201cfc431d90baca1ad0a397cf (127.8.86.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:05:41.185218 29017 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:05:41.185585 28974 raft_consensus.cc:1050] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf: attempting to promote NON_VOTER 8484181c4ece483c8b408817fab1a917 to VOTER
I20240502 14:05:41.185978 28974 consensus_queue.cc:237] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } }
I20240502 14:05:41.187368 15700 consensus_queue.cc:237] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: false } }
I20240502 14:05:41.187708 15569 raft_consensus.cc:1261] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:05:41.188194 29199 consensus_queue.cc:1035] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 14:05:41.188341 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:41.188606 15832 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:05:41.188668 15569 raft_consensus.cc:1261] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:05:41.188954 15700 raft_consensus.cc:1261] T 28e371c1a999431499e09479e8f5b44b P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:05:41.189025 28904 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:05:41.189132 29003 consensus_queue.cc:1035] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:05:41.189373 29199 consensus_queue.cc:1035] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:05:41.190763 28991 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: false } } }
I20240502 14:05:41.191095 28747 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:05:41.191264 15569 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: false } } }
I20240502 14:05:41.192260 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 76f07464d9bd49a9b1b9d58f6a180cf8 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:05:41.193044 15831 raft_consensus.cc:2932] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: false } } }
I20240502 14:05:41.193109 15702 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:05:41.193200 15832 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:05:41.194891 15428 catalog_manager.cc:5484] T 76f07464d9bd49a9b1b9d58f6a180cf8 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 10 to 11, VOTER 2254b3ee3db647598d781e467584f78d (127.8.86.136) evicted. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:05:41.196476 15424 catalog_manager.cc:5484] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 9 to 10, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:05:41.197464 29042 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:05:41.198122 15700 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:05:41.198246 15569 raft_consensus.cc:1203] T 03dd735113ab4ea6bdb696b11a4efd7f P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240502 14:05:41.199666 15960 consensus_queue.cc:237] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } }
I20240502 14:05:41.199644 15569 raft_consensus.cc:2932] T 28e371c1a999431499e09479e8f5b44b P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:05:41.201588 15700 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:05:41.201962 15832 raft_consensus.cc:1261] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:05:41.201923 29026 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:05:41.201944 15426 catalog_manager.cc:5484] T 28e371c1a999431499e09479e8f5b44b P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 9 to 10, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:05:41.202258 29026 consensus_queue.cc:1035] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:05:41.203544 29017 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:05:41.203975 15700 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:05:41.206233 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9391f16ae2d247ec8a71e5c971cdda71 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:05:41.206209 15832 raft_consensus.cc:2932] T 9391f16ae2d247ec8a71e5c971cdda71 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:05:41.206467 15428 catalog_manager.cc:5484] T 9391f16ae2d247ec8a71e5c971cdda71 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 10 to 11, VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:05:41.212046 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:05:41.240625 28952 consensus_queue.cc:579] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Leader has been unable to successfully communicate with peer 3c3658c946994e2ba9f41b64aa327d25 for more than 300 seconds (300.159s)
W20240502 14:05:41.240998 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:05:41.243763 15569 consensus_queue.cc:237] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } }
I20240502 14:05:41.245368 15960 raft_consensus.cc:1261] T bb9f100600a246708457280d91826fce P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:05:41.245772 28952 consensus_queue.cc:1035] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 14:05:41.246271 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:41.246417 15832 raft_consensus.cc:1261] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:05:41.246884 28952 consensus_queue.cc:1035] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:05:41.248760 28968 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } } }
I20240502 14:05:41.249202 15832 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } } }
I20240502 14:05:41.250442 15415 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bb9f100600a246708457280d91826fce with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:05:41.251971 15429 catalog_manager.cc:5484] T bb9f100600a246708457280d91826fce P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 9 to 10, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New cstate: current_term: 1 leader_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } } }
I20240502 14:05:41.252141 15960 raft_consensus.cc:2932] T bb9f100600a246708457280d91826fce P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } } }
I20240502 14:05:41.263392 15700 raft_consensus.cc:1203] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 14:05:41.348706 28952 raft_consensus.cc:1050] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf: attempting to promote NON_VOTER 82d7b7201cfc431d90baca1ad0a397cf to VOTER
I20240502 14:05:41.349143 28952 consensus_queue.cc:237] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: false } }
I20240502 14:05:41.350788 15832 raft_consensus.cc:1261] T d81f14b7c1d1468196d0fe6d67fe93a7 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEARNER]: Refusing update from remote peer 1eb0e3a6c782469fa8110f6bb6a585cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:05:41.351203 28968 consensus_queue.cc:1035] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 14:05:41.351342 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:05:41.351440 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:41.384797 28904 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:41.433699 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:41.447197 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:05:41.513046 15569 raft_consensus.cc:1203] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
W20240502 14:05:41.616212 15413 catalog_manager.cc:4573] TS 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet bb9f100600a246708457280d91826fce: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:05:41.656150 29041 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:05:41.679178 28904 consensus_queue.cc:579] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Leader has been unable to successfully communicate with peer f688370715454d6e814a78fcf85ac5c3 for more than 300 seconds (300.300s)
I20240502 14:05:41.681888 15700 consensus_queue.cc:237] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } }
I20240502 14:05:41.683269 15960 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:05:41.683269 15832 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:05:41.683691 29176 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:05:41.683840 28904 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:05:41.685706 28991 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } } }
I20240502 14:05:41.686259 15960 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } } }
I20240502 14:05:41.686275 15832 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } } }
I20240502 14:05:41.687474 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 48fbc670ec504d26a11bf1b9fbd78981 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:05:41.689241 15429 catalog_manager.cc:5484] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf reported cstate change: config changed from index 10 to 11, VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } } }
I20240502 14:05:41.692833 15700 consensus_queue.cc:237] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } }
I20240502 14:05:41.694154 15960 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:05:41.694283 15832 raft_consensus.cc:1261] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:05:41.694732 29003 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 14:05:41.694913 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:41.695041 28991 consensus_queue.cc:1035] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:05:41.696403 28904 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:05:41.697676 15960 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:05:41.698836 15832 raft_consensus.cc:2932] T 48fbc670ec504d26a11bf1b9fbd78981 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } } }
I20240502 14:05:41.699340 15428 catalog_manager.cc:5484] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 11 to 12, NON_VOTER 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135) added. New cstate: current_term: 1 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: NON_VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:41.700310 15413 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 48fbc670ec504d26a11bf1b9fbd78981 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:05:41.700783 28747 raft_consensus.cc:1050] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd: attempting to promote NON_VOTER 8484181c4ece483c8b408817fab1a917 to VOTER
I20240502 14:05:41.701174 28747 consensus_queue.cc:237] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } }
I20240502 14:05:41.702900 15700 raft_consensus.cc:1261] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:05:41.702961 15832 raft_consensus.cc:1261] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:05:41.703895 29017 consensus_queue.cc:1035] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:05:41.704027 29026 consensus_queue.cc:1035] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 14:05:41.704396 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:41.706205 29026 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:05:41.706705 15700 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:05:41.708531 15832 raft_consensus.cc:2932] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:05:41.709466 15429 catalog_manager.cc:5484] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 9 to 10, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:05:41.817207 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:41.821172 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:05:41.862146 28747 raft_consensus.cc:1050] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd: attempting to promote NON_VOTER 8484181c4ece483c8b408817fab1a917 to VOTER
I20240502 14:05:41.862589 28747 consensus_queue.cc:237] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } }
I20240502 14:05:41.864212 15700 raft_consensus.cc:1261] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [term 1 LEARNER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:05:41.864368 15832 raft_consensus.cc:1261] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Refusing update from remote peer 4d0f8f311e834d81bd54b9f8e7923bcd: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:05:41.864641 29026 consensus_queue.cc:1035] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:05:41.865059 29178 consensus_queue.cc:1035] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 14:05:41.865763 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:41.867036 29026 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:05:41.867504 15700 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 8484181c4ece483c8b408817fab1a917 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:05:41.868896 15832 raft_consensus.cc:2932] T 2d92e5885bf94c3e8e0e4dee40017699 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } } }
I20240502 14:05:41.870321 15426 catalog_manager.cc:5484] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 9 to 10, 8484181c4ece483c8b408817fab1a917 (127.8.86.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:05:41.873525 28974 consensus_queue.cc:579] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Leader has been unable to successfully communicate with peer f688370715454d6e814a78fcf85ac5c3 for more than 300 seconds (300.337s)
W20240502 14:05:41.894060 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 14:05:41.904011 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:05:41.909591 29042 consensus_queue.cc:579] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Leader has been unable to successfully communicate with peer 2254b3ee3db647598d781e467584f78d for more than 300 seconds (300.373s)
W20240502 14:05:41.910917 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:05:41.962661 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:41.978991 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:05:41.988608 29003 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:42.017907 15637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 151 similar messages]
I20240502 14:05:42.051784 15418 maintenance_manager.cc:419] P d9f1f4817ca9494da13bfe388be84b9d: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 11217 bytes on disk
I20240502 14:05:42.054222 15417 maintenance_manager.cc:643] P d9f1f4817ca9494da13bfe388be84b9d: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.001s sys 0.001s Metrics: {"dirs.queue_time_us":190,"thread_start_us":104,"threads_started":1}
I20240502 14:05:42.073769 29042 raft_consensus.cc:1050] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf: attempting to promote NON_VOTER 1eb0e3a6c782469fa8110f6bb6a585cf to VOTER
I20240502 14:05:42.074193 29042 consensus_queue.cc:237] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: false } }
I20240502 14:05:42.075534 15569 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 LEARNER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 14:05:42.075634 15960 raft_consensus.cc:1261] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Refusing update from remote peer 82d7b7201cfc431d90baca1ad0a397cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 14:05:42.076076 29241 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:05:42.076212 29042 consensus_queue.cc:1035] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240502 14:05:42.076385 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:42.078011 29241 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: false } } }
I20240502 14:05:42.078476 15960 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: false } } }
I20240502 14:05:42.081650 15428 catalog_manager.cc:5484] T 2886dfbec1e141198b0cf7a435fac498 P 4d0f8f311e834d81bd54b9f8e7923bcd reported cstate change: config changed from index 12 to 13, 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82d7b7201cfc431d90baca1ad0a397cf" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: false } } }
I20240502 14:05:42.082818 15569 raft_consensus.cc:2932] T 2886dfbec1e141198b0cf7a435fac498 P 1eb0e3a6c782469fa8110f6bb6a585cf [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: VOTER last_known_addr { host: "127.8.86.131" port: 41531 } } peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: false } } }
W20240502 14:05:42.103261 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:42.142771 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:42.231189 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:05:42.266849 28991 consensus_queue.cc:579] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [LEADER]: Leader has been unable to successfully communicate with peer f688370715454d6e814a78fcf85ac5c3 for more than 300 seconds (300.257s)
I20240502 14:05:42.269501 15700 consensus_queue.cc:237] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } }
I20240502 14:05:42.270695 15960 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 3 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 14:05:42.271349 28991 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 14:05:42.271620 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:42.273286 29250 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:05:42.273801 15960 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } }
I20240502 14:05:42.275106 15413 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e76bd0ecd3e4c168e141fb7984af3ec with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:05:42.276351 15429 catalog_manager.cc:5484] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 8 to 10, VOTER f688370715454d6e814a78fcf85ac5c3 (127.8.86.133) evicted. New cstate: current_term: 3 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:05:42.278409 29042 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:05:42.279522 15700 consensus_queue.cc:237] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: true } }
W20240502 14:05:42.281147 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129:36237): Couldn't send request to peer 1eb0e3a6c782469fa8110f6bb6a585cf. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:42.281152 15960 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 3 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
W20240502 14:05:42.281236 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:42.281625 29250 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:05:42.283077 28991 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, NON_VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: true } } }
I20240502 14:05:42.283533 15960 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, NON_VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: true } } }
I20240502 14:05:42.284365 15413 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6e76bd0ecd3e4c168e141fb7984af3ec with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:05:42.285461 15428 catalog_manager.cc:5484] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 10 to 11, NON_VOTER 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) added. New cstate: current_term: 3 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:05:42.350245 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 14:05:42.403285 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:42.412691 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:42.415719 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:42.420655 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:42.426229 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:42.434260 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:05:42.448277 29003 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:42.516644 15506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 14:05:42.517279 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:42.529503 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 14:05:42.538936 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:42.551407 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:42.631424 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6e76bd0ecd3e4c168e141fb7984af3ec: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:42.633841 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:05:42.642191 29026 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:42.642406 29026 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:05:42.642869 29026 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:05:42.643862 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:42.643949 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:05:42.644038 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:05:42.644186 29026 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:05:42.671319 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:05:42.682193 29266 ts_tablet_manager.cc:902] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: Initiating tablet copy from peer 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261)
I20240502 14:05:42.682718 29266 tablet_copy_client.cc:308] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet copy: Beginning tablet copy session from remote peer at address 127.8.86.130:42261
I20240502 14:05:42.683194 15721 tablet_copy_service.cc:140] P 8484181c4ece483c8b408817fab1a917: Received BeginTabletCopySession request for tablet 6e76bd0ecd3e4c168e141fb7984af3ec from peer 1eb0e3a6c782469fa8110f6bb6a585cf ({username='slave'} at 127.8.86.129:32983)
I20240502 14:05:42.683297 15721 tablet_copy_service.cc:161] P 8484181c4ece483c8b408817fab1a917: Beginning new tablet copy session on tablet 6e76bd0ecd3e4c168e141fb7984af3ec from peer 1eb0e3a6c782469fa8110f6bb6a585cf at {username='slave'} at 127.8.86.129:32983: session id = 1eb0e3a6c782469fa8110f6bb6a585cf-6e76bd0ecd3e4c168e141fb7984af3ec
I20240502 14:05:42.684233 15721 tablet_copy_source_session.cc:205] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:05:42.685036 29266 ts_tablet_manager.cc:1875] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:05:42.687683 29266 ts_tablet_manager.cc:1888] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240502 14:05:42.687932 29266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e76bd0ecd3e4c168e141fb7984af3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:05:42.689627 29266 tablet_copy_client.cc:783] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet copy: Starting download of 5 data blocks...
W20240502 14:05:42.693001 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:05:42.694553 29266 tablet_copy_client.cc:647] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet copy: Starting download of 1 WAL segments...
I20240502 14:05:42.695806 29266 tablet_copy_client.cc:523] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:05:42.697005 29266 tablet_bootstrap.cc:492] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap starting.
I20240502 14:05:42.705456 29266 tablet_bootstrap.cc:492] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:05:42.705698 29266 tablet_bootstrap.cc:492] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: Bootstrap complete.
I20240502 14:05:42.705883 29266 ts_tablet_manager.cc:1367] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20240502 14:05:42.706348 29266 raft_consensus.cc:348] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: true } }
I20240502 14:05:42.706462 29266 raft_consensus.cc:729] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 1eb0e3a6c782469fa8110f6bb6a585cf, State: Initialized, Role: LEARNER
I20240502 14:05:42.706547 29266 consensus_queue.cc:260] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 3.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: true } }
I20240502 14:05:42.706817 29266 ts_tablet_manager.cc:1398] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 14:05:42.707360 15721 tablet_copy_service.cc:342] P 8484181c4ece483c8b408817fab1a917: Request end of tablet copy session 1eb0e3a6c782469fa8110f6bb6a585cf-6e76bd0ecd3e4c168e141fb7984af3ec received from {username='slave'} at 127.8.86.129:32983
I20240502 14:05:42.707458 15721 tablet_copy_service.cc:433] P 8484181c4ece483c8b408817fab1a917: ending tablet copy session 1eb0e3a6c782469fa8110f6bb6a585cf-6e76bd0ecd3e4c168e141fb7984af3ec on tablet 6e76bd0ecd3e4c168e141fb7984af3ec with peer 1eb0e3a6c782469fa8110f6bb6a585cf
W20240502 14:05:42.721591 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:42.735222 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:42.882077 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:05:42.901139 29277 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:05:42.924469 29003 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:05:42.935639 15618 maintenance_manager.cc:419] P 1eb0e3a6c782469fa8110f6bb6a585cf: Scheduling UndoDeltaBlockGCOp(6e76bd0ecd3e4c168e141fb7984af3ec): 4586 bytes on disk
I20240502 14:05:42.936152 15508 maintenance_manager.cc:643] P 1eb0e3a6c782469fa8110f6bb6a585cf: UndoDeltaBlockGCOp(6e76bd0ecd3e4c168e141fb7984af3ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
W20240502 14:05:42.951756 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:42.981659 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:05:43.049690 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:43.095496 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 14:05:43.187256 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:43.223543 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:05:43.252722 15569 raft_consensus.cc:1203] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 3 LEARNER]: Deduplicated request from leader. Original: 3.10->[3.11-3.11] Dedup: 3.11->[]
W20240502 14:05:43.255100 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:43.354615 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:05:43.439679 29290 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:43.457355 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:05:43.465096 29297 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:43.549192 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240502 14:05:43.661226 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:43.768580 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:05:43.772117 29003 raft_consensus.cc:1050] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917: attempting to promote NON_VOTER 1eb0e3a6c782469fa8110f6bb6a585cf to VOTER
I20240502 14:05:43.772482 29003 consensus_queue.cc:237] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 8, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: false } }
I20240502 14:05:43.773897 15960 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 3 FOLLOWER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 14:05:43.773952 15569 raft_consensus.cc:1261] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 3 LEARNER]: Refusing update from remote peer 8484181c4ece483c8b408817fab1a917: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 14:05:43.774418 29305 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:05:43.774703 29003 consensus_queue.cc:1035] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 14:05:43.774838 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:05:43.776376 29305 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 11 to 12, 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: false } } }
I20240502 14:05:43.776825 15569 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 1eb0e3a6c782469fa8110f6bb6a585cf [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: false } } }
I20240502 14:05:43.778246 15960 raft_consensus.cc:2932] T 6e76bd0ecd3e4c168e141fb7984af3ec P 4d0f8f311e834d81bd54b9f8e7923bcd [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: false } } }
I20240502 14:05:43.780017 15428 catalog_manager.cc:5484] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 reported cstate change: config changed from index 11 to 12, 1eb0e3a6c782469fa8110f6bb6a585cf (127.8.86.129) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "8484181c4ece483c8b408817fab1a917" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb0e3a6c782469fa8110f6bb6a585cf" member_type: VOTER last_known_addr { host: "127.8.86.129" port: 36237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:05:43.792873 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:43.803532 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:05:43.828207 29305 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:43.828271 15413 catalog_manager.cc:4573] TS 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b0cd21dc897b41aca9118be6df48cc0c: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:43.836640 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 14:05:43.915580 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:43.980335 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:44.001212 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 14:05:44.038188 29297 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:44.149406 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:05:44.212029 29304 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:44.222129 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:44.222170 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:44.245038 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:44.302302 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:44.351768 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:05:44.382019 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240502 14:05:44.428877 29297 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:44.488359 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:05:44.503387 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:44.508435 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:05:44.574800 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:44.588737 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:44.620712 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:44.626653 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:05:44.738539 29338 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:44.749294 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:44.763887 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:44.857910 15414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 14:05:44.858639 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2886dfbec1e141198b0cf7a435fac498: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:44.864547 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:05:44.922766 29277 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:44.968891 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:44.999558 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 14:05:45.003491 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:45.043092 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:45.089490 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:45.121384 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:45.132169 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:45.142701 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:45.173481 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:45.179142 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:45.182098 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 14:05:45.321990 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:05:45.359261 29338 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:05:45.372658 29277 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:45.384907 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:45.392248 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:45.412034 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:45.426813 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 14:05:45.449798 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:45.497949 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:45.518466 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:45.563657 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:45.708772 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 14:05:45.764544 29305 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:45.781520 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 14:05:45.785332 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:45.807394 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:05:45.944350 29277 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:45.947827 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 48fbc670ec504d26a11bf1b9fbd78981: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:46.060189 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:46.108397 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:46.128002 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:05:46.139915 29305 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:46.319758 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 14:05:46.377163 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:46.398552 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 14:05:46.412937 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:46.490252 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:46.509130 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240502 14:05:46.546862 29277 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:05:46.625455 29338 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:46.653744 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:46.667018 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 14:05:46.754218 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:46.807232 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:46.907696 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:46.911633 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:46.944916 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 14:05:46.951575 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:46.981305 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:47.012267 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:47.012892 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:47.039206 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240502 14:05:47.054435 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:05:47.065903 29390 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:47.099273 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:47.145911 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:47.196890 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 14:05:47.201601 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:05:47.234324 29409 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:47.292695 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 14:05:47.321332 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:05:47.449884 29390 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:47.520947 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:47.536002 15506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 14:05:47.545105 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:47.567276 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:47.590399 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:47.598243 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:47.614480 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 14:05:47.615195 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:47.643678 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:47.669023 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:05:47.675146 29409 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:47.697407 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:47.810655 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:47.878490 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:47.931783 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:05:47.933431 29390 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:47.954463 15413 catalog_manager.cc:4573] TS 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b0cd21dc897b41aca9118be6df48cc0c: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:05:48.014256 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:48.025530 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:48.045519 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:48.160689 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:48.173875 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:48.200917 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240502 14:05:48.267271 29440 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:05:48.375918 29390 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:48.385483 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:48.473330 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 14:05:48.485513 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:05:48.547326 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:48.584251 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 14:05:48.609587 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:48.626423 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:05:48.673318 29409 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:48.681886 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 14:05:48.731492 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:48.733292 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240502 14:05:48.790095 29277 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:48.800891 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:48.998441 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2886dfbec1e141198b0cf7a435fac498: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:49.001209 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:49.042408 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:49.060511 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:49.067610 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:49.087378 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 14:05:49.196475 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:05:49.234937 29390 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:05:49.256124 29409 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:49.262392 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:49.310866 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 14:05:49.347714 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:49.353688 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:49.468483 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:49.474138 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:49.489298 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:49.519515 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 14:05:49.532951 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:49.583418 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:49.651830 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 14:05:49.679052 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:05:49.793460 29479 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:05:49.827286 29390 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:49.886086 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 14:05:49.886745 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:49.907594 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:49.909001 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:49.921128 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:50.017999 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:50.081054 15414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 16 similar messages]
W20240502 14:05:50.081740 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 48fbc670ec504d26a11bf1b9fbd78981: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:50.104156 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:50.109833 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:50.158077 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:50.199061 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 14:05:50.227474 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:50.331214 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:50.345293 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:50.360455 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:05:50.375669 29277 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:50.378065 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:05:50.390509 29479 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:50.404040 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:50.426613 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:50.553244 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:50.709367 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:50.758824 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:50.783891 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 14:05:50.896080 29515 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:50.911689 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:05:50.924363 29479 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:50.960245 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:51.049901 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:05:51.063633 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 14:05:51.098394 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:51.120877 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:51.131554 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 14:05:51.151974 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:51.202991 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:51.206029 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 14:05:51.282544 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240502 14:05:51.393078 29277 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:51.434113 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:51.462275 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:05:51.462672 29517 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:51.543531 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:51.543706 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 14:05:51.548118 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:51.593001 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:51.724794 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:51.781004 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:51.833580 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240502 14:05:51.834314 29277 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:51.879370 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:51.917503 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 14:05:51.981891 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:51.985847 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:51.992908 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:52.053086 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 152 similar messages]
I20240502 14:05:52.053505 29545 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:52.131590 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:52.180523 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:52.194881 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:52.282330 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:52.289587 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:05:52.291041 29515 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:52.300719 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:52.315726 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 14:05:52.358798 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:52.438223 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:52.448235 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 14:05:52.487352 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:52.554849 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 14:05:52.578023 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:05:52.648501 29545 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:52.695115 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:05:52.708216 29515 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:52.782521 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:52.783668 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 14:05:52.807776 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:53.005756 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:53.015092 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:53.045379 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:53.116964 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:05:53.178812 29545 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:53.196206 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:05:53.210990 29515 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:53.220237 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:53.247895 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:53.292425 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:53.381150 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:53.414017 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:53.521205 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:53.567116 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:53.595873 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:05:53.606819 29517 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:05:53.618005 29277 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:53.645409 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 14:05:53.720084 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:05:53.727383 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 14:05:53.772147 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 14:05:53.773250 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 14:05:53.783512 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:53.816373 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:53.822831 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 14:05:53.924922 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:53.955116 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:54.120118 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:54.122268 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240502 14:05:54.130744 29277 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:54.222205 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:05:54.230540 29607 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:54.237432 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:54.260468 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 14:05:54.282759 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:54.424531 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:54.456117 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 14:05:54.513723 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:54.563897 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:05:54.601295 29277 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:54.634344 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:54.687990 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:54.703703 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240502 14:05:54.728211 29607 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:54.733551 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:54.756824 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:54.832628 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:54.837184 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:54.896414 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:54.913022 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:05:54.915458 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:54.944043 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:54.972175 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:55.079460 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240502 14:05:55.133962 29607 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:05:55.161386 29277 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:55.195576 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:55.223842 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:55.388852 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:55.391737 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 14:05:55.481541 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:55.572202 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:05:55.633710 29600 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:55.674588 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:55.722072 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:55.724965 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:55.746762 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:05:55.753123 29607 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:55.787019 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:55.832823 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 14:05:55.833474 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:55.835110 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:55.837034 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:55.884799 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:55.958833 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:56.063604 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:05:56.098491 29277 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:56.100322 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 14:05:56.169288 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:56.186666 15413 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240502 14:05:56.187368 15413 catalog_manager.cc:4573] TS 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b0cd21dc897b41aca9118be6df48cc0c: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
I20240502 14:05:56.189442 29649 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:56.194707 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 14:05:56.217221 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:56.334412 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:56.413617 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 14:05:56.426956 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 14:05:56.436941 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:56.464215 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:56.499747 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:56.606189 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240502 14:05:56.664558 29277 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:05:56.737763 29649 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:56.740624 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:56.784770 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 14:05:56.813048 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:56.880592 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:56.907547 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:56.912954 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:56.937327 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:56.958806 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:57.048316 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:57.094555 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240502 14:05:57.095250 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240502 14:05:57.132319 29678 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:57.185312 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:57.194187 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:57.215108 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:57.221434 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2886dfbec1e141198b0cf7a435fac498: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:05:57.251147 29679 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:57.296202 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:57.335371 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:57.369261 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:57.463505 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 14:05:57.489182 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:57.515477 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:57.525487 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 14:05:57.538776 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:57.565114 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:57.567492 15506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 141 similar messages]
I20240502 14:05:57.623332 29277 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:05:57.821979 29686 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:05:57.822130 29686 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:05:57.822525 29679 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:05:57.822690 29686 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:05:57.823643 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:05:57.823737 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:05:57.823817 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:05:57.824014 29686 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:05:57.832810 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:57.847028 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:05:58.009658 29277 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:58.025744 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 14:05:58.062036 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:58.062127 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:58.073151 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:58.122736 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:05:58.256314 29649 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:58.260515 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:58.306222 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 48fbc670ec504d26a11bf1b9fbd78981: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:05:58.367508 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:58.412048 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 14:05:58.424983 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:58.441857 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:58.448966 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:58.460660 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:58.484135 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240502 14:05:58.503569 29678 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:58.505043 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:58.590106 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:58.621064 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 14:05:58.632529 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:58.643988 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:05:58.682178 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 14:05:58.740478 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 103 similar messages]
I20240502 14:05:58.761102 29649 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:58.805552 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:05:58.984277 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:58.992970 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:59.050643 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:05:59.119210 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:05:59.121354 29748 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:59.182147 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:59.246238 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:05:59.329038 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 14:05:59.346256 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 14:05:59.376469 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:05:59.384208 29754 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:59.419679 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:05:59.426716 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:05:59.452387 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:59.551309 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:05:59.561837 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:05:59.586427 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:05:59.588291 29748 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:59.660624 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 14:05:59.718088 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:05:59.735705 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:05:59.739403 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 14:05:59.784260 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:05:59.803097 29754 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:05:59.829911 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:05:59.841042 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:05:59.899001 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:05:59.940469 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:00.002323 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240502 14:06:00.100073 29678 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:00.166569 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:00.210687 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:06:00.216818 29754 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:00.250504 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:00.491379 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:06:00.574777 29678 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:00.574826 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:00.583734 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:06:00.614734 29649 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:00.628679 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 14:06:00.705571 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:00.720602 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:00.832754 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:00.858420 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 14:06:00.884915 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:00.953838 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:00.962913 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 14:06:00.982374 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:06:01.010633 29748 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:06:01.012379 29649 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:01.032843 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:01.034920 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:06:01.041913 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:01.071997 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 14:06:01.086241 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:01.114809 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:01.149752 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 14:06:01.191110 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:01.199712 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:01.221231 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 14:06:01.384537 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:01.480203 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:01.488911 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:01.496140 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:06:01.566222 29809 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:06:01.576205 29649 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:01.723295 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:01.775727 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 14:06:01.948887 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 14:06:01.990337 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:02.047788 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:02.054227 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:02.060354 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:06:02.069007 29810 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:02.109480 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:02.121330 15637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240502 14:06:02.126720 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:02.166683 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 14:06:02.168915 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:02.186460 29809 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:02.216192 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:02.261080 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:02.275305 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:02.297950 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:02.323347 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 14:06:02.432641 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 14:06:02.445897 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:02.473197 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:02.510542 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:02.570818 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 137 similar messages]
I20240502 14:06:02.673488 29845 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:02.682647 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:02.730273 29846 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:02.732911 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:02.738340 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:02.740442 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:02.859740 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:02.959440 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:02.991312 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
I20240502 14:06:03.141204 29845 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:03.141709 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:03.176890 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:06:03.229736 29846 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:03.246305 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:03.316974 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:03.385051 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:03.398356 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:03.413089 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:03.421800 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:03.439183 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 14:06:03.446084 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:03.542212 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:03.608772 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:03.670354 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:03.672343 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:03.675371 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 14:06:03.718186 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
I20240502 14:06:03.726884 29846 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:06:03.754622 29875 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:03.760455 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 14:06:03.792807 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 14:06:03.830793 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:04.043167 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:04.088769 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 14:06:04.099475 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:04.129990 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:04.244518 29887 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:04.251543 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:04.253981 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:04.305964 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240502 14:06:04.376147 29875 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:04.377851 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:04.390246 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:04.468317 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:04.506729 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:04.535894 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:04.587000 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 14:06:04.707911 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:04.715834 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:04.756711 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:06:04.793130 29845 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:04.818578 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:06:04.830396 29887 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:04.840304 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 14:06:04.890043 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:04.928128 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:04.952612 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:05.033735 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 14:06:05.073849 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:05.095538 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:05.148682 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:05.159794 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:05.202373 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:05.210582 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:06:05.282907 29887 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:05.331823 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:05.369463 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:06:05.401234 29924 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:05.446475 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 14:06:05.489940 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:05.533656 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:05.613518 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 14:06:05.718281 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:05.738039 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:06:05.810451 29887 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:05.835681 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:05.884685 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:05.895911 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:06:05.903609 29924 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:05.965694 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:05.970240 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:05.999846 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:06.016460 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 14:06:06.017544 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:06:06.213199 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 14:06:06.223707 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:06.250663 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:06:06.284720 29924 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:06.291055 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:06.358614 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240502 14:06:06.415345 29887 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:06.487284 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:06.526844 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:06.538604 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:06.543896 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:06.721093 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:06.768559 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:06.777639 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 14:06:06.796790 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:06.825883 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:06:06.845002 29957 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:06:06.856021 29924 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:06.991497 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:06.994318 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:06.996883 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 14:06:07.103327 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:07.103592 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:07.202312 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:07.205770 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:07.209805 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240502 14:06:07.262516 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 14:06:07.307550 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
I20240502 14:06:07.356153 29845 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:07.395437 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:06:07.430871 29986 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:07.448824 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:07.471467 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:07.556933 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:07.592286 15506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240502 14:06:07.623412 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:07.659682 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:07.687273 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:07.690454 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:07.711583 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:07.774132 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:07.779109 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:07.783334 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 14:06:07.854007 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:06:07.941339 29845 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:07.972927 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:06:07.975346 29986 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:08.019651 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 14:06:08.081629 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:08.099682 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:08.241632 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:08.343829 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:08.395522 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 14:06:08.433138 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:06:08.522833 29924 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:08.575706 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240502 14:06:08.585618 30016 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:08.589318 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:08.590628 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:08.613824 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:08.685408 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 14:06:08.692632 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:08.717782 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:08.764225 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:06:08.771556 29985 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:08.771704 29985 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:06:08.772119 29985 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:06:08.773152 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:06:08.773262 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:06:08.773351 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:06:08.773583 29985 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:06:08.814698 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:06:08.838181 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:08.929230 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:08.950109 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 14:06:08.958598 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:06:09.022766 29845 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:09.152441 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:06:09.178587 30016 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:09.251955 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:09.255152 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:09.278420 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:09.336545 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:09.360095 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 14:06:09.409291 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 14:06:09.416230 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:06:09.477979 29924 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:09.478953 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240502 14:06:09.483219 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:09.561908 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:09.673213 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:09.692174 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:06:09.757941 30051 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:09.782563 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:09.785622 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240502 14:06:09.878938 29924 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:09.897949 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:09.933895 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:09.942052 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:09.955822 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:09.994777 15413 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: ChangeConfig RPC to 127.8.86.130:42261 timed out after 30.000s (SENT) [suppressed 10 similar messages]
W20240502 14:06:09.996261 15413 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a7981c3f46d8483caae82fbf2296e762 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }.
Pending config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } }
W20240502 14:06:10.041587 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:10.072814 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:10.161406 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:10.166321 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:10.167021 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:10.181492 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:06:10.202342 30051 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:10.207567 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:10.246356 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:10.257447 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:10.379807 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:06:10.421023 29924 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:10.421420 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:10.427186 29985 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:10.427309 29985 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:06:10.427747 29985 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:06:10.428694 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:06:10.428829 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:06:10.428893 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:06:10.429001 29985 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:06:10.589633 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 14:06:10.629014 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 14:06:10.744611 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:06:10.761090 30051 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:10.764448 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:10.777369 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:10.801896 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:10.934062 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:10.962247 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:10.972687 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240502 14:06:11.038018 30087 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:11.039264 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:11.042694 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240502 14:06:11.130393 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 14:06:11.130426 15413 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a7981c3f46d8483caae82fbf2296e762 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }.
Pending config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } }
W20240502 14:06:11.166111 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 14:06:11.171751 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:11.210237 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:11.215633 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 14:06:11.247351 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:06:11.267622 30094 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:11.348017 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:11.459261 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:06:11.618388 30087 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:11.636866 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:11.674135 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 14:06:11.781915 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:11.782640 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:11.803179 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:11.803953 30112 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:11.814517 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:11.826124 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:11.858206 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 14:06:11.895515 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:12.100142 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:06:12.108618 30069 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:12.141842 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:12.158438 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 14:06:12.205245 15413 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a7981c3f46d8483caae82fbf2296e762 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }.
Pending config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } }
W20240502 14:06:12.205567 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:12.219673 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240502 14:06:12.239542 15637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 146 similar messages]
W20240502 14:06:12.240164 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:12.279961 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:12.357898 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:06:12.401084 30112 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:12.459887 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:12.488588 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:12.517912 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:12.576788 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:12.582860 15413 catalog_manager.cc:4573] TS 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b0cd21dc897b41aca9118be6df48cc0c: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:06:12.587481 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:12.635003 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:06:12.642270 30069 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:12.647886 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240502 14:06:12.669580 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:12.709628 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:12.733597 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:12.744808 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:12.744915 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:06:12.787783 30130 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:12.876560 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:12.902252 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:13.109690 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:13.190917 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:13.219336 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:13.220607 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:06:13.229285 30069 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:13.231305 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:13.290206 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 14:06:13.302182 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240502 14:06:13.318601 30130 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:13.327061 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:13.589984 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 14:06:13.649070 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:06:13.654799 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2886dfbec1e141198b0cf7a435fac498: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:06:13.660542 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:13.676198 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:06:13.686870 30087 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:13.732563 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 14:06:13.784243 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:13.795060 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 14:06:13.844095 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:13.862325 30155 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:13.880065 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 14:06:13.933857 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:13.955026 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240502 14:06:14.063527 30087 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:14.064134 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:14.080647 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 14:06:14.102458 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:14.172443 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:14.242823 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:14.243930 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:06:14.250394 30130 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:14.277804 15413 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a7981c3f46d8483caae82fbf2296e762 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }.
Pending config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } }
W20240502 14:06:14.390950 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 14:06:14.438746 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 14:06:14.462044 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:14.469483 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:14.513258 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:06:14.516958 30087 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:14.596602 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:14.633616 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:14.666940 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:14.725373 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 48fbc670ec504d26a11bf1b9fbd78981: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:06:14.776371 30198 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:14.789991 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:14.809585 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:14.810441 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:14.901901 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:14.941543 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240502 14:06:14.955763 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:14.974985 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:15.085537 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:06:15.094671 30069 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:15.128108 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:15.164453 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:15.174062 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:15.185305 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:15.268349 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:15.286741 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:15.287302 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:06:15.300740 30215 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:15.324189 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:15.353123 15414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 14:06:15.360306 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:06:15.631487 30087 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:15.636808 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:15.643834 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:15.779027 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:15.786494 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 14:06:15.788761 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:06:15.805227 30230 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:15.820700 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:15.823581 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 14:06:15.977316 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 14:06:16.043933 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:06:16.061080 30087 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:16.114094 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:06:16.202491 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 14:06:16.207201 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:16.218334 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:16.251652 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:16.286859 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 14:06:16.296784 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:16.354853 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:16.377987 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240502 14:06:16.381733 30230 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:16.486227 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:06:16.486639 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:16.608021 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:16.611374 30087 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:16.612743 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:16.680852 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 14:06:16.704643 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:16.736090 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240502 14:06:16.760646 30246 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:16.790704 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 14:06:16.839550 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:16.874042 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:16.898198 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:16.962280 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:17.087509 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:17.167411 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:17.177567 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:17.203580 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:06:17.213124 30087 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:17.220623 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:17.252931 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240502 14:06:17.278103 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:17.278192 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:17.313525 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240502 14:06:17.379498 30274 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:17.392202 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:17.414638 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:17.476944 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:17.547493 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:17.596807 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:17.601329 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:17.611780 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:06:17.706099 30069 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:17.706956 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 14:06:17.744863 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:17.745337 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:17.798643 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:06:17.855134 30274 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:17.918128 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:17.931824 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:17.975525 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:18.044646 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:18.164391 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:18.192350 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240502 14:06:18.253563 30069 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:18.334275 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240502 14:06:18.352154 30246 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:18.389422 15413 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a7981c3f46d8483caae82fbf2296e762 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }.
Pending config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } }
W20240502 14:06:18.506506 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:18.518080 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:18.530907 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 14:06:18.558163 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:18.740841 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:18.748876 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:06:18.756237 30295 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:18.798045 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 14:06:18.828305 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:18.839707 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240502 14:06:18.840796 30274 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:18.859418 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:18.901634 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:18.914368 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:18.925668 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:06:18.983064 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:19.006778 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:19.107682 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:06:19.163343 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 14:06:19.173388 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 14:06:19.193185 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:19.206264 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:06:19.246246 30295 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:19.271916 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:06:19.275729 30246 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:19.369513 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:19.369627 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:19.370354 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:19.396692 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 14:06:19.474673 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:19.541229 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 14:06:19.620332 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:19.640425 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:06:19.697281 30246 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:19.697541 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:19.718608 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:06:19.764336 30295 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:19.777428 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 14:06:19.821573 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:19.911379 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:19.933063 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:19.969530 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:20.003449 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:20.098670 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:20.101883 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:06:20.230952 30348 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:20.240962 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:20.277345 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:06:20.329694 30295 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:20.373561 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:20.383137 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:20.467224 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:20.494837 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:20.517751 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:20.530385 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:20.577370 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:20.702627 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240502 14:06:20.810878 30246 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:20.824324 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:06:20.828469 30349 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:20.914855 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:21.097311 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 14:06:21.117945 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:06:21.130868 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 14:06:21.131433 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:21.132495 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:21.211395 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240502 14:06:21.220284 30348 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:21.253700 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 14:06:21.289511 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:21.386524 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:21.387060 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:06:21.423657 30349 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:21.459174 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:21.499109 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 14:06:21.524689 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:06:21.593613 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:21.598104 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 14:06:21.626883 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:21.654014 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:21.663329 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:21.697819 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:21.724460 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:06:21.798846 30348 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:21.809135 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 14:06:21.823115 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:21.924311 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:21.950917 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:21.972364 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:06:22.029170 30349 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:22.032575 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 14:06:22.069650 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:22.082371 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:22.216631 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:06:22.223630 30348 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:22.226933 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:22.293159 15637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240502 14:06:22.368834 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:22.378271 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:22.384524 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:22.392261 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:22.439831 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240502 14:06:22.440507 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:22.530120 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:06:22.548998 30349 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:22.632781 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:22.655129 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:22.664772 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:22.678488 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:22.712682 15506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 14:06:22.787972 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:06:22.806636 30420 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:22.824769 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:22.922585 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:22.940063 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:22.953856 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:22.955109 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:06:23.044862 30414 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:06:23.318521 30436 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:23.334312 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:23.341881 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:23.362396 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 14:06:23.432090 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 14:06:23.644881 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240502 14:06:23.660985 30442 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:23.688417 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:23.695516 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 14:06:23.704717 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 14:06:23.742681 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:23.853896 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 14:06:23.870929 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:23.911738 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:23.921556 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
I20240502 14:06:23.923133 30436 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:23.931711 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:06:23.933971 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 14:06:23.964625 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:24.000233 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:24.038192 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:06:24.075191 30455 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:24.101593 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:24.132095 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:24.182894 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:24.185055 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 14:06:24.219135 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:24.276291 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:24.354535 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:24.367269 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:24.402375 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:24.442961 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:24.455729 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:06:24.527832 30466 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:24.531893 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:06:24.602910 30455 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:24.633389 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 14:06:24.648490 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:24.681577 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:24.722498 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:24.828003 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:24.864854 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 14:06:24.957374 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:25.052897 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:25.064381 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:06:25.066028 30482 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:25.073697 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:06:25.080829 30455 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:25.087002 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:25.087019 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:25.191395 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:06:25.272897 29985 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:25.273061 29985 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:06:25.273512 29985 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:06:25.274564 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:06:25.274757 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:06:25.274829 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:06:25.274955 29985 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:06:25.311942 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:25.331980 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:25.460919 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:06:25.524431 30466 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:25.546218 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:06:25.549795 30455 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:25.550729 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:25.571794 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:25.616698 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:25.620817 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 14:06:25.799228 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 14:06:25.879843 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 14:06:25.915381 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:06:25.963176 30482 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:25.990721 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:26.069416 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:26.084621 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:06:26.144326 30455 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:26.192380 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 14:06:26.221396 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:26.231598 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 14:06:26.279502 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:26.295444 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 14:06:26.301726 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:26.321059 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:06:26.363693 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:26.412341 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 14:06:26.425944 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:06:26.463274 30466 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:26.493958 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:26.493984 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 14:06:26.603859 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:26.616940 15413 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a7981c3f46d8483caae82fbf2296e762 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }.
Pending config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } }
W20240502 14:06:26.685233 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 14:06:26.699453 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:06:26.702960 30455 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:26.745553 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:26.830080 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:26.837148 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:26.877084 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:26.896327 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:26.920069 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:26.946146 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:26.972916 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:27.001914 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:27.023319 30466 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:27.053654 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:06:27.177242 30531 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:27.227233 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:27.369088 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 14:06:27.381937 15637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240502 14:06:27.382618 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:27.496202 30554 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:27.497211 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:27.508478 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:27.510634 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:27.557891 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:27.558230 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:27.598508 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 14:06:27.628973 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:27.629817 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:27.650494 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:06:27.658246 30455 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:27.731554 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240502 14:06:27.792606 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:27.803231 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:27.930800 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:27.964761 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:27.977277 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240502 14:06:28.056460 30531 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:06:28.083936 30570 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:28.128793 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:28.174954 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:28.224453 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:28.357915 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 14:06:28.424764 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:28.455031 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
I20240502 14:06:28.568140 30537 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:28.594902 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:28.613323 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:06:28.617806 30455 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:28.618309 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:28.628994 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 14:06:28.726182 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:28.741313 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:28.764860 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:06:28.770473 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 14:06:28.836295 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:28.843261 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:28.929265 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240502 14:06:29.002401 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 14:06:29.006044 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:06:29.073576 30537 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:29.111197 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:29.123123 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240502 14:06:29.140537 30531 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:29.152228 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:29.226117 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:29.250150 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:29.270926 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:29.288900 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:29.330866 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 14:06:29.407622 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:29.470336 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:29.496724 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:29.527774 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:06:29.530679 30537 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:29.610580 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:29.631146 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:29.692489 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:06:29.739193 30613 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:29.778388 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:29.820528 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:29.877285 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:29.905398 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240502 14:06:29.914600 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:29.918684 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:30.000803 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:06:30.037019 30621 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:30.085475 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:30.131376 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:30.132117 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:06:30.174625 30613 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:30.225539 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:30.268303 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:30.301342 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 14:06:30.431602 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:30.475136 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:06:30.530491 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:06:30.538301 30633 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:30.538331 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:30.589749 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240502 14:06:30.670519 30531 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:30.677897 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:30.774217 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:30.886207 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 14:06:30.912501 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:06:31.003123 30621 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:31.004230 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 14:06:31.092156 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240502 14:06:31.093988 30613 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:31.099287 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240502 14:06:31.100186 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:31.135746 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:31.279328 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 14:06:31.314077 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:31.387918 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:31.388468 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:31.410413 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:31.439888 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:06:31.440646 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:31.473596 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:06:31.520392 30613 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:06:31.590505 30668 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:31.598927 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:31.664927 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240502 14:06:31.698454 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:31.702415 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:31.757347 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:31.809257 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 14:06:31.850517 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:31.928637 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:31.941725 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:31.967453 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:31.985775 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:32.029291 30613 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:32.149739 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:06:32.159515 30668 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:32.169507 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:32.174652 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:32.217334 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:32.244065 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 14:06:32.272687 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:32.305927 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:32.309474 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:32.394548 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240502 14:06:32.395320 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:32.412585 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:06:32.437059 30613 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:32.506923 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:32.512581 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:32.526036 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:32.585193 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:32.679546 30668 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:32.693074 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:32.775485 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240502 14:06:32.815285 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240502 14:06:32.844949 30700 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:32.854596 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:32.987442 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:33.022936 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:33.104532 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:33.180892 30668 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:33.283993 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240502 14:06:33.304217 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:33.314601 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240502 14:06:33.381338 30719 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:33.418649 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:33.486215 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:33.486456 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:33.568691 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 14:06:33.606225 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240502 14:06:33.651088 30713 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:33.682026 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:33.715261 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:33.761739 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:06:33.784000 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:33.801920 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240502 14:06:33.983135 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240502 14:06:33.998217 30719 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:34.039108 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:34.066344 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240502 14:06:34.067507 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:34.131374 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:06:34.139251 30713 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:34.158946 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:34.205440 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:34.226094 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:34.240969 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:34.305701 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:34.309108 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:34.323657 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:34.330029 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:34.337625 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 14:06:34.369782 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:34.378764 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:06:34.417809 30700 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:34.467286 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240502 14:06:34.569314 30668 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:34.632181 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:34.702975 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 14:06:34.716467 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:34.742921 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:34.791857 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:34.811944 30719 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:34.835637 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:34.864974 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:34.940508 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:35.006449 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:35.029903 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:35.050586 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:35.069280 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:35.097568 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240502 14:06:35.142320 30668 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:35.169348 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:35.209483 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:35.213856 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:35.254063 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:06:35.360051 30719 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:35.438853 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:35.549086 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 14:06:35.570415 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:06:35.683192 30771 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:35.687919 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:35.724463 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:35.737735 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:35.775269 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240502 14:06:35.896299 30719 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:35.918731 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:35.988705 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 14:06:36.126142 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:06:36.163367 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 14:06:36.196767 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:06:36.221305 30771 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:36.224282 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:36.234328 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:36.235195 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:36.326129 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:36.431090 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:06:36.445104 30807 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:36.536631 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 14:06:36.606346 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 14:06:36.607040 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:36.620654 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:36.628526 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 14:06:36.631736 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 14:06:36.641425 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:36.654050 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:36.657675 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:36.729838 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:36.745653 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:36.762017 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:06:36.827598 30800 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:36.840849 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:36.871925 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:36.951192 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 14:06:36.993705 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:06:37.047369 30807 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:37.070785 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:37.148387 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:37.191229 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:37.287773 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:06:37.308813 30771 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:37.372457 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:37.382495 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:37.399487 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240502 14:06:37.423796 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:37.424479 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 14:06:37.432211 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:06:37.436287 30719 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:37.442083 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:37.499904 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:37.541435 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 14:06:37.580614 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:37.638794 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:37.651813 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:37.702067 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:06:37.728792 30800 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:37.834656 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 14:06:37.891835 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240502 14:06:37.934867 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:06:37.963984 30845 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:38.129999 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:38.146795 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:38.149039 30771 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:38.213864 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:38.272936 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:38.314668 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 14:06:38.352978 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:38.399850 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:38.469555 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:06:38.535498 30845 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:38.553107 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:38.580160 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 14:06:38.621564 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:06:38.623050 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:38.643193 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:38.679244 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:38.684422 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240502 14:06:38.692065 30771 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:06:38.810851 30640 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:38.810992 30640 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:06:38.811466 30640 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:06:38.812510 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:06:38.812592 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:06:38.812652 15897 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:06:38.812775 30640 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:06:38.927906 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:06:38.992787 30862 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:39.077140 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:06:39.077775 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:39.083771 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:39.092864 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:39.120091 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:39.163791 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:06:39.164484 30771 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:39.233315 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:39.235314 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:39.292575 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 14:06:39.320372 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:39.342895 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 14:06:39.345733 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 14:06:39.410014 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:39.413233 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:39.499091 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 14:06:39.512445 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:39.545975 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:39.599246 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:06:39.610605 30893 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:39.643616 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:06:39.679510 30899 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:39.693349 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:39.768155 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:39.827850 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 14:06:39.853569 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 14:06:39.906791 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:39.933652 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:39.960204 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:39.979008 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:40.065418 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:06:40.096218 30899 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:40.108438 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:06:40.232169 30893 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:40.239126 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:40.288444 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:40.320906 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 14:06:40.422255 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:40.484030 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:40.486996 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:06:40.577338 30899 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:06:40.656921 30912 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:40.730079 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:40.847345 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:40.887671 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:40.924491 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:06:40.935554 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:06:40.971419 30771 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:40.994493 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:40.998458 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:41.007170 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 14:06:41.047623 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 14:06:41.056900 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:41.104763 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:41.233788 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:41.238884 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:06:41.241142 30912 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:41.287292 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:41.343714 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:41.378917 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:06:41.450328 30899 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:41.525521 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 14:06:41.598999 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 14:06:41.623970 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 14:06:41.641302 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:06:41.745744 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:41.762755 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:41.777623 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:06:41.790352 30924 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:41.804200 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:41.826102 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:41.883595 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:41.962710 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240502 14:06:41.993321 30899 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:42.031733 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:42.042133 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:42.085307 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:42.087461 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:42.095475 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:42.100463 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:42.102144 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 14:06:42.133782 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:42.139084 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 14:06:42.191735 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:06:42.243359 30924 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:42.271709 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 14:06:42.283444 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:42.355664 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:42.412227 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240502 14:06:42.415237 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:06:42.466931 30964 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:42.540442 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:42.541291 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:42.592722 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:42.688450 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 14:06:42.748996 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:42.842756 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:42.846168 30924 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:42.931919 15506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 14:06:42.957304 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:06:42.985755 30991 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:43.001438 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:43.026741 15413 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a7981c3f46d8483caae82fbf2296e762 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }.
Pending config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } }
W20240502 14:06:43.044988 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:43.104771 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 14:06:43.311655 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:43.314975 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:06:43.355796 30924 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:43.433961 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:43.447536 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:43.448063 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:43.455305 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:06:43.487947 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240502 14:06:43.511080 30991 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:43.561740 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:43.671407 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:43.686183 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:43.702214 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:43.711776 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:43.738677 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:43.764073 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:43.777981 30957 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:43.778115 30957 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:06:43.778520 30957 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:06:43.779533 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:06:43.779723 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:06:43.779788 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:06:43.779927 30957 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:06:43.865432 30924 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:43.883266 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 14:06:43.891700 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240502 14:06:43.894860 30991 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:43.908299 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:44.025624 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 14:06:44.102972 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:44.128542 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240502 14:06:44.180250 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:44.361407 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:06:44.361707 30924 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:44.371450 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:06:44.372148 30991 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:44.396871 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:44.413125 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:44.440066 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:44.468883 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:44.480878 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 14:06:44.497467 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:44.517302 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240502 14:06:44.541348 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:44.622977 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:44.630100 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:44.709499 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:44.722781 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:44.744370 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:44.810720 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240502 14:06:44.847007 30991 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:44.856864 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 14:06:44.907634 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:44.908835 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:06:44.978224 30924 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:45.018985 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:45.021559 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:45.116551 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 14:06:45.139986 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:45.156889 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:45.198505 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:06:45.256310 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:06:45.335676 30991 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:45.397643 15413 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:06:45.398491 15413 catalog_manager.cc:4573] TS 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b0cd21dc897b41aca9118be6df48cc0c: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:06:45.496657 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:06:45.514339 30924 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:45.521524 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:45.536339 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:45.577474 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 14:06:45.634503 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:45.745072 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:45.756134 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:06:45.768883 31071 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:45.769021 31071 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:06:45.769398 31071 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:06:45.770431 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:06:45.770551 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:06:45.770632 15897 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:06:45.770779 31071 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:06:45.900067 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240502 14:06:45.917503 30964 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:45.961568 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:45.969856 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:06:46.025606 30924 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:46.113538 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:46.162561 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:46.191120 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:46.265040 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:46.293283 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240502 14:06:46.330590 30991 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:46.357043 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:06:46.405433 31079 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:46.405436 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:46.408304 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:46.429430 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2886dfbec1e141198b0cf7a435fac498: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:06:46.457479 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:46.541095 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240502 14:06:46.619946 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:46.631827 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 14:06:46.662485 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:46.728189 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240502 14:06:46.728798 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:46.748035 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:46.797884 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:46.824872 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:46.842108 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:06:46.870898 31103 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:46.963310 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 14:06:46.970741 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:06:47.019166 31109 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:47.036650 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 14:06:47.089973 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:47.138588 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:47.140414 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:47.166996 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:47.174649 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:47.192796 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:47.263208 30991 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:47.275200 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 14:06:47.278211 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240502 14:06:47.303117 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:47.380311 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:47.395385 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:47.415064 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240502 14:06:47.429637 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:47.434382 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:06:47.451702 31079 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:47.458009 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240502 14:06:47.479962 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:47.537914 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 48fbc670ec504d26a11bf1b9fbd78981: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:06:47.623270 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:47.687379 30991 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:47.741456 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:47.850271 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:47.854032 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:47.864948 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:47.976575 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:48.007577 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240502 14:06:48.028790 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:48.034529 31079 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:48.105039 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240502 14:06:48.178968 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:06:48.301429 31145 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:48.362017 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:06:48.410940 31136 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:48.443235 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:06:48.453591 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:48.497231 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:48.534814 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 14:06:48.582795 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
I20240502 14:06:48.588070 31101 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:48.588232 31101 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:06:48.588689 31101 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:06:48.589651 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:06:48.589779 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:06:48.589856 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:06:48.589993 31101 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:06:48.612063 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:48.621173 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:48.658723 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:48.662823 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:06:48.798978 31145 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:48.839548 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:48.856341 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:48.862378 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:06:48.927414 31136 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:49.110550 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 14:06:49.145254 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 14:06:49.148088 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:49.159818 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 14:06:49.192395 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:49.242785 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:49.253952 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:06:49.283752 30991 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:49.322145 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:49.349025 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:49.427726 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240502 14:06:49.434823 31180 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:49.435576 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 14:06:49.465651 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:49.528117 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240502 14:06:49.557958 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:49.596336 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:49.602494 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 14:06:49.606734 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:49.633455 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:49.663863 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:49.669449 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240502 14:06:49.695425 31145 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:49.738839 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:49.743425 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:49.793428 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:49.805728 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:49.908066 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:49.929613 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:06:49.942812 31180 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:50.079905 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:50.096901 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:50.101763 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:06:50.151432 30991 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:50.204926 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:50.276116 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:50.374186 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 14:06:50.377225 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:50.441135 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:06:50.508153 31209 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:50.513088 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:50.540799 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:50.641059 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:50.646811 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:50.746098 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:06:50.771883 30991 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:50.858466 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:50.913676 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:50.938323 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240502 14:06:50.951587 31209 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:50.963745 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:51.001441 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:06:51.112869 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 14:06:51.144634 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:51.265970 31145 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:51.345402 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:51.386632 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:51.414772 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 14:06:51.427426 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:06:51.474139 31209 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:51.482182 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:51.567178 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:51.668543 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:51.668586 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:51.700838 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:06:51.722949 30991 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:51.750228 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240502 14:06:51.777879 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 14:06:51.803344 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:51.807854 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:06:51.889391 31239 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:51.890166 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 14:06:51.892995 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:51.900964 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:51.928680 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 14:06:52.062479 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:52.103111 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 14:06:52.110040 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:52.111405 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:52.132161 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:52.150161 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:06:52.193333 30991 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:52.199645 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:52.231767 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 14:06:52.291931 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 14:06:52.325215 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:52.338809 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:06:52.341425 31209 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:52.361686 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:52.428504 15637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240502 14:06:52.495821 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:52.503585 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:52.529273 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:52.715935 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 14:06:52.724979 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:52.726788 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:06:52.731206 31209 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:52.749579 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:52.757480 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:52.796139 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:52.797245 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:06:52.811698 31274 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:52.837888 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:53.023149 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 14:06:53.034477 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:53.104952 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:53.107884 31239 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:53.195608 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:06:53.220791 31274 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:53.231874 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:53.316349 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:06:53.385506 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 14:06:53.447883 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:53.477619 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 14:06:53.488762 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:06:53.520175 31209 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:53.646768 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:53.747280 31274 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:53.795763 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:53.921124 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:53.924273 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:06:53.958045 31209 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:53.967229 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:53.969467 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:54.010622 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 14:06:54.116938 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:54.147848 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:54.164700 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:06:54.165344 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:54.172263 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:54.214854 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:54.264397 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
I20240502 14:06:54.290892 31274 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:54.351123 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:06:54.364053 31239 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:54.436493 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 14:06:54.519841 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:54.521602 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 14:06:54.548249 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:54.554948 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:54.591627 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:54.600950 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:54.665366 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:54.731371 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 14:06:54.736541 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:54.804600 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:06:54.805380 31332 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:54.817541 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 14:06:54.823060 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:54.839918 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:54.843231 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240502 14:06:54.919723 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:06:54.975857 31239 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:54.988483 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:55.013778 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:55.027204 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:55.031023 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:55.171331 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:55.190668 31332 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:55.232769 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 14:06:55.284539 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:55.349990 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:55.398015 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:55.404178 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:06:55.481470 31239 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:55.536088 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:55.536650 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:55.577803 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:55.645079 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:55.774427 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:06:55.788763 31359 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:55.853529 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
I20240502 14:06:55.931627 31209 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:55.978801 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:06:56.070533 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:56.119048 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 14:06:56.153681 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:06:56.225889 31359 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:56.372897 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:56.421039 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:56.432832 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
I20240502 14:06:56.481806 31239 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:56.556932 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:56.581873 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:06:56.614251 31321 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:56.614434 31321 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:06:56.614898 31321 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:06:56.615865 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:06:56.616739 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:06:56.616819 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:06:56.617000 31321 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:06:56.621842 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:56.674466 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 14:06:56.681166 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:56.691869 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:56.699827 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:06:56.709683 31359 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:56.734478 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:56.747910 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 14:06:56.785013 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 14:06:56.785686 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:56.789633 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:56.906944 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:06:56.930045 31239 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:56.970229 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 14:06:57.058557 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 14:06:57.074674 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:57.089205 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:06:57.097920 31332 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:57.114933 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:57.116345 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:57.147032 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 14:06:57.186013 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:57.186955 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:57.187007 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:57.207157 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:57.248678 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:57.287823 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240502 14:06:57.308341 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:57.322950 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:06:57.428315 31239 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:57.484800 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:57.506253 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 152 similar messages]
I20240502 14:06:57.543402 31332 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:57.647601 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:57.715471 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:57.790206 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 14:06:57.809437 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:57.841229 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:57.875950 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:57.906344 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:57.957305 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:06:58.020709 31239 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:58.058177 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:58.058993 15505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240502 14:06:58.059666 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:06:58.109618 31332 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:58.137180 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:58.143093 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:58.155633 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:58.204071 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 14:06:58.258273 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:06:58.304849 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:06:58.406637 31413 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:06:58.485477 31419 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:58.605124 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 14:06:58.697925 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:58.749545 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:58.817940 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
I20240502 14:06:58.937741 31413 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:58.952075 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:06:59.009701 31445 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:59.018765 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 14:06:59.079547 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:59.109380 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:06:59.114203 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:59.154510 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 14:06:59.159511 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:59.229094 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:59.283064 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:06:59.291517 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240502 14:06:59.292383 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:59.331636 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:06:59.336616 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:06:59.386205 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:06:59.386598 31445 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:59.398648 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 14:06:59.416075 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:06:59.444990 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:06:59.535321 31461 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:59.536669 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:06:59.550505 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:06:59.575299 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:06:59.656073 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 14:06:59.694104 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:06:59.762746 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 14:06:59.768179 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:06:59.790966 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:06:59.806684 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:06:59.836407 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:06:59.863773 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 14:06:59.876986 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:06:59.958019 31474 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:06:59.979085 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:00.028764 31461 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:00.089193 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:00.105585 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:00.171476 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:00.187784 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:07:00.293541 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 14:07:00.300668 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:00.389149 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:00.408016 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:07:00.430759 31445 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:00.436101 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:00.496503 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:00.510327 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 14:07:00.544883 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:07:00.591573 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:00.602810 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:00.653820 31492 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:00.778999 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:00.831844 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:07:00.920966 31474 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:01.098003 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:01.209110 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
I20240502 14:07:01.232893 31509 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:01.251798 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:01.326841 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 14:07:01.385180 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 14:07:01.423520 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:07:01.472856 31516 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:01.547581 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:01.574774 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:01.606112 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:07:01.679890 31509 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:01.740691 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:01.816632 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:01.832696 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:01.863329 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:01.925182 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:07:01.958197 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:01.991416 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 14:07:02.003101 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:02.020244 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 14:07:02.039637 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:02.070649 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:02.071033 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:02.071470 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:02.092252 31516 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:02.107266 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:02.107576 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:07:02.134753 31509 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:02.148078 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:02.163472 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:02.196668 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:02.214715 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:02.227046 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:02.370940 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 14:07:02.383739 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 14:07:02.436010 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:02.452073 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:02.513075 15637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240502 14:07:02.526852 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:02.575692 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:07:02.594424 31516 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:02.611825 31509 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:02.673655 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:02.689220 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:02.701689 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 14:07:02.849603 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 14:07:02.883886 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:02.927651 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:02.944371 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 14:07:02.969187 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:02.993127 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:03.026786 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:07:03.040827 31545 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:03.095187 31509 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:03.100553 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:03.150224 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:03.193109 15505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 14:07:03.193984 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:03.288376 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:03.461432 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:07:03.487208 31516 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:03.571125 31492 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:03.582357 31503 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:07:03.582530 31503 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:07:03.583019 31503 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:07:03.584089 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:07:03.584179 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:07:03.584261 15897 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:07:03.584400 31503 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:07:03.707949 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 14:07:03.768321 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:03.824643 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:03.859522 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 14:07:03.887423 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 14:07:03.943578 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:03.948760 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:07:03.976536 31509 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:04.015729 31545 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:04.085759 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 14:07:04.096822 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:04.285213 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:04.290550 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:04.347654 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:07:04.348279 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:04.365137 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:04.401082 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 14:07:04.405797 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:07:04.417846 31545 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:04.443162 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:04.451854 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:04.466650 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:07:04.501037 31509 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:04.557556 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:04.606781 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:04.610095 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:04.630209 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:04.646782 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:04.650781 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:04.661955 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:04.684060 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:04.808635 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:04.879774 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:04.885959 31603 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:04.901283 31516 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:04.902798 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:04.958096 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:04.962390 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:04.992942 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:05.014281 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 14:07:05.067807 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:05.099242 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 14:07:05.186991 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
I20240502 14:07:05.335320 31603 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:05.335398 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 14:07:05.384608 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:07:05.384905 31516 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:05.388033 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 14:07:05.437009 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:05.461767 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:05.462116 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:05.553516 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:05.559943 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:05.602586 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:05.742558 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:05.780076 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:07:05.884804 31545 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:05.910866 31635 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:05.931658 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 14:07:06.027328 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:06.101727 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:06.239895 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:06.270632 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240502 14:07:06.301429 31635 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:06.319543 31545 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:06.432082 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:06.440989 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:06.459414 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 14:07:06.552050 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:06.567339 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:06.704747 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240502 14:07:06.725129 31509 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:06.870334 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:06.881989 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:07:06.884429 31664 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:06.948066 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:06.975112 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:06.987136 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:07.027487 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:07.027899 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:07.059670 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:07.066980 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240502 14:07:07.067668 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:07.068184 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:07.160287 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:07.164228 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:07.190897 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:07.252171 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:07.261620 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:07.276468 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 14:07:07.277244 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:07:07.336675 31509 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:07.367605 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:07:07.400975 31545 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:07.411911 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:07.430609 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:07.459599 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:07.519986 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:07.544838 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:07.566488 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240502 14:07:07.586364 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 14:07:07.654858 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:07.709537 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 14:07:07.759193 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 14:07:07.859560 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:07.866199 31694 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:07.869220 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:07.890755 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:07.898173 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 14:07:07.917690 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:07:07.927166 31545 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:07.941573 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 14:07:07.953575 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:07.993352 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:08.053186 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:08.057286 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:08.098917 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:08.303148 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240502 14:07:08.364857 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:08.416584 31677 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:08.438619 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:07:08.473021 31545 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:08.475559 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 14:07:08.814754 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:08.842485 31694 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:08.849367 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:08.914330 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:08.949385 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240502 14:07:08.951069 31545 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:09.004585 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 14:07:09.005666 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 14:07:09.086620 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:09.149200 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:09.152989 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:09.252074 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:09.298193 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:07:09.301185 31694 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:09.338511 31664 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:09.414709 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:09.416606 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:09.420403 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 14:07:09.441008 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:09.462576 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:09.490432 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:09.532110 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:09.538808 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:09.566449 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:09.635843 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 14:07:09.643703 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:09.647238 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:09.656543 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:09.680367 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:09.694576 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:09.716749 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:07:09.729187 31677 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:09.809348 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:09.873862 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:07:09.880417 31750 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:09.883356 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:09.926111 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:09.974720 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:10.020602 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:10.087705 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:10.088575 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:10.174631 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:10.177029 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:10.261324 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 14:07:10.299266 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240502 14:07:10.323073 31761 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:10.343026 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 14:07:10.352241 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240502 14:07:10.416143 31762 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:10.454371 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 14:07:10.553782 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:10.573267 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:10.591099 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:10.611343 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:10.651559 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:10.673224 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:07:10.710490 31677 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:10.970574 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:07:11.000830 31779 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:11.024668 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:11.063877 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:11.090085 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 14:07:11.197293 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240502 14:07:11.201967 31761 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:11.444516 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 14:07:11.469470 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:07:11.507050 31778 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:11.531294 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:11.597012 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:11.601061 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:11.693400 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:11.701681 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:11.712255 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:11.713178 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:11.763142 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240502 14:07:11.764233 31803 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:11.769331 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:11.788928 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:11.821923 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:11.873471 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:11.873512 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:07:11.890978 31778 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:12.037142 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:12.079320 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:07:12.080036 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:12.087836 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:12.104507 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 14:07:12.104987 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:12.158515 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:12.189375 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:12.201629 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:12.236069 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:07:12.312088 31803 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:12.335932 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:12.348887 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:07:12.350831 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:12.418272 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:12.424480 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:07:12.502569 31823 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:12.586055 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240502 14:07:12.588866 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:12.590579 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 14:07:12.603081 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:12.675350 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:12.724447 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:12.731580 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240502 14:07:12.852927 31803 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:12.952209 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 14:07:12.972993 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:07:12.994134 31778 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:13.091786 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 14:07:13.121548 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:13.182417 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:13.183862 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:13.190093 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:13.204746 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:13.275202 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 14:07:13.316171 15506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 141 similar messages]
I20240502 14:07:13.411621 31845 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:13.500097 31778 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:13.590626 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 14:07:13.609650 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:13.622848 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:13.641353 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 14:07:13.752777 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:13.786804 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:13.858794 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:07:13.887228 31845 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:13.930210 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240502 14:07:13.993674 31778 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:14.020695 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:14.087241 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:14.114043 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:14.192683 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:14.243741 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:14.247485 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:07:14.269937 31845 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:14.281308 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:14.285413 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:14.287987 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 14:07:14.299253 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:14.402714 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:14.427167 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:07:14.435678 31851 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:14.463047 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:14.471581 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 14:07:14.490157 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:14.553231 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:14.600996 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:14.632310 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:14.649782 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:14.667034 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:07:14.675697 31803 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:14.690336 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:14.712220 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:14.722992 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 14:07:14.903318 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:07:14.925146 31778 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:14.961831 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:07:14.981377 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:15.005122 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:15.077405 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:15.107995 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 14:07:15.113303 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:15.154023 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:07:15.220436 31845 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:15.270485 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:15.362699 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:15.434633 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:15.445564 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 14:07:15.523803 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240502 14:07:15.546098 31778 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:15.566454 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 14:07:15.678701 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:07:15.752096 31845 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:15.752393 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:15.803368 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240502 14:07:15.804276 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:15.811013 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:15.833719 15413 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a7981c3f46d8483caae82fbf2296e762 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }.
Pending config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } }
W20240502 14:07:15.855849 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:16.020524 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:16.103199 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:07:16.117623 31778 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:16.133885 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
I20240502 14:07:16.133917 31845 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:16.278228 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:16.391151 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:16.413489 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:16.487643 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:16.550943 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
I20240502 14:07:16.556707 31778 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:16.701925 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:16.710363 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:07:16.717809 31845 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:16.725349 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:16.726472 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:16.742201 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:16.784147 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 14:07:16.790827 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:16.830353 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:16.862360 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 14:07:16.922582 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:16.955242 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:16.978422 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:17.003405 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:17.016449 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:17.059321 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:17.072355 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:17.075515 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:07:17.079424 31778 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:17.138980 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 14:07:17.218155 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 14:07:17.218824 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:17.248483 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:17.282341 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:17.314198 31845 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:17.343174 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:17.378335 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:17.425266 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 14:07:17.476861 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:07:17.493934 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:17.604591 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:17.606199 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:17.620290 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 152 similar messages]
I20240502 14:07:17.691423 31778 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:17.717803 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:17.781585 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:17.803045 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:17.829015 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240502 14:07:17.877902 31803 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:17.902662 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 14:07:17.931783 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 14:07:18.017973 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:18.089282 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:18.152527 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:07:18.209069 31978 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:18.324832 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240502 14:07:18.359493 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:18.371428 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:18.396901 31803 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:18.416253 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:18.426278 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:18.436626 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:18.449399 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240502 14:07:18.586475 31778 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:18.679163 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 14:07:18.746793 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:18.769285 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:18.779421 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 14:07:18.823712 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:07:18.852320 31803 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:18.977213 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:19.094398 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:19.114755 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:19.121273 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
I20240502 14:07:19.162812 31778 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:19.164914 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:19.193837 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:19.203645 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:07:19.248030 31803 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:19.249058 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:19.264570 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:19.299612 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:19.477860 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:19.525259 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:07:19.539963 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:19.540356 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:19.541112 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 14:07:19.551299 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:19.585310 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:19.596750 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:19.606875 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240502 14:07:19.645186 31978 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:19.649519 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:19.663316 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:19.687968 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:07:19.701364 31986 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:19.791828 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:19.812439 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:19.869060 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:19.978488 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 14:07:19.978529 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:19.999166 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:19.999647 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:07:20.174561 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:07:20.209378 31978 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:20.210841 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:20.220863 31803 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:20.246816 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:20.253526 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:20.301234 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 14:07:20.307719 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 14:07:20.338752 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:20.495316 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:20.560844 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 14:07:20.690610 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:20.731549 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:07:20.774387 31803 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:20.794195 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:20.798177 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
I20240502 14:07:20.814697 31978 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:20.957562 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:20.977362 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:21.054944 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:21.073107 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 14:07:21.147156 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:21.184744 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:07:21.204614 31803 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:21.247736 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:21.309463 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240502 14:07:21.343937 31978 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:21.403784 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:21.451328 31985 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:07:21.451470 31985 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:07:21.452023 31985 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:07:21.452988 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:07:21.453076 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:07:21.453135 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:07:21.453270 31985 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:07:21.512393 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:21.605060 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:21.642536 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:07:21.643664 31803 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:21.711288 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:21.777348 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 14:07:21.788446 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:21.822368 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:21.877795 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:21.902261 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:07:21.913605 32079 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:21.943781 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:21.989893 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 14:07:21.996703 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240502 14:07:22.014775 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:22.113344 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:07:22.115860 31803 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:22.153999 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:22.164472 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:22.187089 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:22.242486 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:07:22.243171 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:22.252368 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:22.269130 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:22.314527 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:22.349653 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:07:22.365840 31978 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:22.434723 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:22.460707 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 14:07:22.472887 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:22.474331 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:22.529637 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 14:07:22.535799 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:22.574072 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:22.698079 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240502 14:07:22.707923 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240502 14:07:22.728137 31803 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:22.765089 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:07:22.872089 32102 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:22.899130 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:07:22.903151 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:23.076951 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:23.113222 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:23.140931 31803 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:23.173115 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 14:07:23.200470 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:07:23.249553 32102 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:23.267446 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 14:07:23.350971 15505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 14:07:23.415650 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:23.422088 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:23.487785 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:23.492833 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:23.595405 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:23.615186 31803 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:23.688274 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:23.689519 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:07:23.803635 32102 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:23.837538 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:23.848158 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:23.865639 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240502 14:07:23.908398 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 14:07:23.975378 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:23.977489 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:23.999408 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:24.126776 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:07:24.180966 31986 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:24.229154 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:24.245597 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:24.305476 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:24.305562 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 14:07:24.336895 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:07:24.365447 32102 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:24.397181 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:24.437245 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240502 14:07:24.514149 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240502 14:07:24.515281 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:24.517484 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:24.573964 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:24.630697 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 103 similar messages]
I20240502 14:07:24.695221 31986 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:24.704591 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:24.707809 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:24.760105 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:24.776057 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:07:24.783926 32102 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:24.789779 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:24.804024 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:24.858078 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:24.869424 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:24.892768 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 14:07:24.912743 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:25.001129 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:25.055413 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:25.084062 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:07:25.109774 31986 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:25.136862 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 14:07:25.158725 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:07:25.201917 32102 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:25.252586 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:25.336751 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 14:07:25.394502 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:07:25.564970 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:25.592844 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240502 14:07:25.594760 31803 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:25.718195 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:25.730429 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:07:25.750093 32102 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:25.758846 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240502 14:07:25.844128 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:25.873733 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:25.976954 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:07:25.999908 31803 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:26.070521 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:26.074688 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:07:26.130328 32181 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:26.149819 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:26.169270 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:26.242118 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:26.298039 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 14:07:26.399657 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:26.458849 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:26.499908 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:26.514627 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240502 14:07:26.545645 32181 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:26.566809 32205 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:26.585953 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:26.647301 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:26.761358 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:26.765987 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 14:07:26.767529 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 14:07:26.824889 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:26.834175 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 14:07:26.858737 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:26.875062 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:26.916219 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:26.918622 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:26.943723 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:07:26.943745 32102 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:27.032492 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:27.059032 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:07:27.073943 32221 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:27.126106 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:27.164582 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:27.177587 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:27.201468 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:27.324007 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:27.374274 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:07:27.416119 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240502 14:07:27.449699 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:27.452742 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:27.467278 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240502 14:07:27.501040 32102 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:27.536686 31985 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:07:27.536834 31985 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:07:27.537310 31985 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:07:27.538497 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:07:27.538596 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:07:27.538684 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:07:27.538838 32233 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:07:27.587446 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:27.655786 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:07:27.685708 32240 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:27.736546 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:27.742846 15637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240502 14:07:27.743530 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:07:27.781126 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:27.815660 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:27.825110 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 14:07:27.830101 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:27.974004 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240502 14:07:28.115134 32232 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:28.170413 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:28.186499 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:07:28.220885 32240 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:28.228381 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:28.367048 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 14:07:28.367146 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:28.373649 15505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 14:07:28.382688 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:28.447016 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:28.450423 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 14:07:28.561408 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:28.623450 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:07:28.705050 32232 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:28.807514 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:07:28.833372 32251 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:28.928423 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:28.931236 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:28.954082 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:28.978025 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:29.008092 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:29.109397 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:29.195842 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:29.209203 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:29.239907 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:29.251029 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 14:07:29.264171 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240502 14:07:29.291848 32240 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:29.298956 32232 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:29.331445 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 14:07:29.340248 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:29.353176 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:29.359320 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 14:07:29.380065 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:29.444075 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:29.500919 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:29.516970 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:29.530895 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:29.556077 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:29.561305 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:29.610342 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:29.657446 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 14:07:29.660334 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:29.777034 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
I20240502 14:07:29.815548 32251 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:29.838497 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:29.864266 32232 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:29.903316 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:29.951939 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:29.988762 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 14:07:30.097848 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:30.120051 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:30.147794 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:30.152877 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 14:07:30.172377 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:30.227643 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:07:30.324095 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240502 14:07:30.341902 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:07:30.407332 32251 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:30.425464 32232 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:30.442837 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:30.457530 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:30.668859 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:07:30.868312 32251 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:30.871433 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:30.893419 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 14:07:30.900069 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:07:30.915927 32297 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:30.946103 32233 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:07:30.946259 32233 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:07:30.946700 32233 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:07:30.947978 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:07:30.948263 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:07:30.948326 15897 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:07:30.948465 32329 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:07:30.972862 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 14:07:31.028064 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:31.044097 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:31.108714 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:31.157850 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:31.160702 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:31.214249 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:07:31.327430 32317 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:31.395375 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:31.443742 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:31.494881 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:31.523351 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:07:31.533747 32232 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:31.567919 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:31.636036 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:31.651294 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:31.721818 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 14:07:31.723927 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:31.759227 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:31.784350 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240502 14:07:31.788640 32317 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:31.890580 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:31.918381 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:31.928577 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 14:07:31.971254 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:32.014865 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:32.031477 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 14:07:32.034448 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:32.055342 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:07:32.090759 32232 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:32.113390 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:32.146029 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:32.178030 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:32.219126 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240502 14:07:32.240223 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:32.263592 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:32.276556 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240502 14:07:32.301066 32336 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:32.450350 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:32.452204 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240502 14:07:32.491653 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:07:32.496829 32297 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:32.579612 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:32.625314 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:32.659013 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:07:32.725204 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:32.734629 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:32.738273 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:32.744632 15637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240502 14:07:32.775441 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:32.778468 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 14:07:32.784772 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240502 14:07:32.921185 32386 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:33.082078 32392 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:33.157716 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:33.279040 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:33.338471 32336 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:33.347944 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:33.379977 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240502 14:07:33.397496 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:33.518985 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:33.582362 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 14:07:33.627776 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:33.634497 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:07:33.638578 32392 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:33.647672 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:33.661527 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:33.666469 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240502 14:07:33.797497 32386 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:33.822932 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:33.886152 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:33.977481 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 14:07:34.008018 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:34.010953 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:34.043377 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:07:34.203845 32421 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:34.204303 32386 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:34.213897 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:34.261005 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:34.267174 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:34.276988 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 14:07:34.342243 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:34.342756 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:34.434599 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:34.475303 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:34.580323 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:07:34.592737 32422 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:34.610461 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 14:07:34.621682 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:34.624239 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:34.650699 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:34.657788 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:34.663033 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 14:07:34.663669 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240502 14:07:34.697458 32392 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:34.724078 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:34.745304 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:34.822978 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 14:07:34.835072 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:34.868880 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:34.873380 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 14:07:34.958680 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:07:34.967846 32386 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:35.037045 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:07:35.103920 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:35.113032 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:35.222533 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:35.225356 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:35.227057 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:35.266337 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:35.270849 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:35.288303 32450 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:35.316377 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240502 14:07:35.513569 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240502 14:07:35.555006 32386 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:35.692392 32392 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:35.738010 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:35.820112 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:35.857885 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:35.915355 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:35.925628 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:35.937404 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:07:35.994148 32461 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:36.093890 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:36.166558 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 14:07:36.254643 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:36.283536 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240502 14:07:36.283931 32479 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:36.301187 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240502 14:07:36.383563 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:36.466984 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:36.468844 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:36.472304 32386 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:36.572958 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:36.677325 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:36.803165 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:07:36.804487 32392 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:36.825668 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240502 14:07:36.848520 32386 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:36.860754 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:36.884528 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:36.904451 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:36.933763 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:36.934876 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:36.950572 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:36.971676 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:36.976572 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 14:07:37.006872 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:37.007565 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:37.016119 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:37.020159 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 14:07:37.044189 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:37.083496 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:37.139847 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 14:07:37.148240 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:07:37.306814 32461 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:37.310777 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:07:37.330155 32506 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:37.366919 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240502 14:07:37.401168 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:37.410354 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:37.448701 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:07:37.473711 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:37.486292 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:37.492125 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 14:07:37.493948 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:37.563640 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:37.572674 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:37.633752 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:37.735976 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:37.750828 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240502 14:07:37.799243 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
I20240502 14:07:37.812755 32392 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:37.877625 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240502 14:07:37.892205 32461 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:38.018131 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:38.063181 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:38.110110 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:38.213841 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:38.250682 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:07:38.320724 32392 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:38.422497 15506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 136 similar messages]
I20240502 14:07:38.452847 32461 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:38.486218 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:38.499131 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:38.726898 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240502 14:07:38.760444 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:07:38.790112 32392 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:38.821931 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:38.843493 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:38.864586 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 14:07:38.901398 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 14:07:38.951927 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:07:39.027645 32541 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:39.052945 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:39.077090 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:39.088153 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:39.172233 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:39.226358 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:39.296705 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:07:39.352277 32392 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:39.405064 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:39.439067 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 14:07:39.453626 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
I20240502 14:07:39.469806 15418 maintenance_manager.cc:419] P d9f1f4817ca9494da13bfe388be84b9d: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033351
I20240502 14:07:39.473060 15417 maintenance_manager.cc:643] P d9f1f4817ca9494da13bfe388be84b9d: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.003s user 0.002s sys 0.000s Metrics: {"bytes_written":2484,"delete_count":0,"lbm_write_time_us":122,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":80,"threads_started":1,"update_count":34,"wal-append.queue_time_us":173}
I20240502 14:07:39.473469 15418 maintenance_manager.cc:419] P d9f1f4817ca9494da13bfe388be84b9d: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.162960
I20240502 14:07:39.475438 32461 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:39.481329 15417 maintenance_manager.cc:643] P d9f1f4817ca9494da13bfe388be84b9d: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.008s user 0.007s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":15862,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":164,"dirs.run_cpu_time_us":74,"dirs.run_wall_time_us":74,"lbm_read_time_us":190,"lbm_reads_lt_1ms":17,"lbm_write_time_us":208,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":124,"threads_started":2,"update_count":34}
W20240502 14:07:39.510303 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240502 14:07:39.520843 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:39.562136 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:39.583191 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:39.587219 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:39.621860 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:39.622009 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240502 14:07:39.638799 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:39.644528 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:39.673342 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 14:07:39.673939 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:39.682451 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:39.704324 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:07:39.758086 32392 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:39.790576 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:39.886348 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 14:07:39.891628 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:39.912930 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:39.915134 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:07:39.944195 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:39.993181 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:40.021761 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:07:40.062341 32541 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:40.119410 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:40.121132 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:40.147289 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:40.255599 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240502 14:07:40.259701 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:07:40.264943 32535 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:40.334277 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 14:07:40.419586 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:07:40.487107 32541 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:40.535950 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:40.703068 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:07:40.796306 32608 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:40.832202 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:40.832293 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:40.948254 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:07:41.093658 32541 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:41.101363 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:41.115506 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:41.202126 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
I20240502 14:07:41.205806 32608 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:41.260334 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 14:07:41.269466 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 14:07:41.274267 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:41.437153 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:41.449939 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:41.474460 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:41.566998 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:41.578940 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240502 14:07:41.590281 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:07:41.628167 32541 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:41.638243 32608 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:41.663342 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 14:07:41.706847 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:41.761045 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:41.816844 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:41.860973 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:41.936408 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 14:07:41.962314 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 14:07:42.042824 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:42.058575 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:42.073263 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:42.073460 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:07:42.102629 32541 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:42.133828 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240502 14:07:42.144980 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:07:42.147119 32646 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:42.159837 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:42.172906 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:42.211266 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:42.214180 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:42.226476 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:42.246129 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:42.369124 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:42.391551 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:42.448745 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:42.474056 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:42.536324 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240502 14:07:42.590332 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 14:07:42.620005 32646 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:42.630352 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:07:42.651924 32541 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:42.681658 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:42.710270 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:07:42.748579 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:42.757758 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240502 14:07:42.762383 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:42.784529 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:42.913903 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 14:07:43.124356 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:07:43.151403 32608 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:43.201633 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:07:43.276700 32541 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:43.310036 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:43.367241 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:43.374020 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:43.443491 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:43.477116 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240502 14:07:43.525727 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:43.574060 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 14:07:43.601269 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
I20240502 14:07:43.666316 32686 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:43.723863 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:43.790753 32541 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:43.839390 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:43.858551 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 14:07:43.918709 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:43.952857 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:43.954839 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 14:07:43.987675 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:44.051154 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:44.070945 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:44.109531 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 14:07:44.132079 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:44.139148 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:07:44.164028 32608 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:44.215257 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:44.229894 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
I20240502 14:07:44.403928 32693 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:44.415095 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:44.432030 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:44.469309 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 14:07:44.472484 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:44.548774 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:44.569907 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:44.633819 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:44.641974 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
I20240502 14:07:44.651414 32686 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:44.707157 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 14:07:44.732051 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:44.732941 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:44.758955 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:44.784305 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:07:44.800621 32541 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:44.821656 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:44.830250 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:44.867235 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240502 14:07:44.974946 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:45.032719 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:07:45.035995 32686 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:45.059257 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:45.072445 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:45.139042 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:45.211397 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:07:45.222919 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:45.233036 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:45.245654 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:07:45.350745 32693 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:45.356003 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:45.442337 15413 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 14:07:45.442999 15413 catalog_manager.cc:4573] TS 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b0cd21dc897b41aca9118be6df48cc0c: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:07:45.554154 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 14:07:45.561712 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:07:45.575191 32686 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:45.742161 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:07:45.775588 32541 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:45.801069 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:45.830797 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:45.831362 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:45.949492 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 14:07:46.049947 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:07:46.077544 32686 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:46.118597 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 14:07:46.120510 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:46.236773 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 14:07:46.308533 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:46.392459 32541 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:46.419800 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:46.524646 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:07:46.527710 32686 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:46.567975 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:46.592631 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 14:07:46.636535 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:46.638697 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:46.684964 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 14:07:46.735090 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:46.736483 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 14:07:46.764178 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22eef55211ca4f65979022d2f4ae829f: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:07:46.792384 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:46.815290 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:46.815747 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:46.824932 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:46.857875 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:46.894940 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:07:46.903594 32686 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:46.906013 309 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:46.963740 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:47.064342 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:47.081218 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:47.122131 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:47.200871 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 14:07:47.220356 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:47.242668 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 14:07:47.247666 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:47.262359 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:47.380522 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:47.390542 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:07:47.427381 32541 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:47.427548 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:47.437817 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:07:47.475817 322 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:47.567368 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240502 14:07:47.584676 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:47.600323 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:07:47.659049 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:47.696722 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:07:47.751699 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:47.774538 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:47.837926 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240502 14:07:47.850683 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:47.866789 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:07:47.904771 309 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:47.988257 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:48.031919 322 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:48.121917 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 14:07:48.133194 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6e76bd0ecd3e4c168e141fb7984af3ec: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:07:48.158506 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:48.350132 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:48.353796 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
I20240502 14:07:48.411195 309 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:48.439009 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:48.467372 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:48.479975 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:48.492659 322 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:48.503904 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 14:07:48.535765 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 14:07:48.565717 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 14:07:48.605342 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:48.718817 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:48.767375 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:07:48.822575 355 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:48.824622 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:48.875355 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:48.894516 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:48.910778 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
I20240502 14:07:48.919555 340 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:48.943686 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:49.155591 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:49.160419 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:49.242554 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:07:49.260316 32329 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:07:49.260449 32329 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:07:49.260912 32329 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:07:49.261814 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:07:49.261922 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:07:49.262010 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:07:49.262143 32329 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:07:49.278177 309 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:49.278611 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:49.306665 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:07:49.317059 322 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:49.318117 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:49.324196 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:49.378225 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 14:07:49.402946 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:49.422571 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240502 14:07:49.434633 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:49.456009 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:49.499008 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:49.500053 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 14:07:49.564360 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 14:07:49.707944 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:49.712220 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 14:07:49.718801 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:49.740270 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:49.761070 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:07:49.772753 355 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:49.775710 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:49.816069 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:07:49.821871 396 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:49.879961 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:49.918550 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:50.033272 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:50.056927 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:50.118316 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:50.118361 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:50.179312 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:07:50.188213 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:50.242066 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
I20240502 14:07:50.317832 396 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:50.325353 309 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:50.363868 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:50.369464 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:50.501798 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 14:07:50.789530 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 14:07:50.836019 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:07:50.857904 396 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:50.863957 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:50.874429 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:50.897792 309 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:50.909881 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:50.957263 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:50.991895 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:51.076169 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:51.077384 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 14:07:51.173516 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:51.229918 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 14:07:51.277829 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:51.367744 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
I20240502 14:07:51.383340 425 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:51.389456 396 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:51.438403 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:51.445602 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:51.476383 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:51.630990 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:51.759686 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:51.778403 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:51.783375 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:07:51.807040 396 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:51.849124 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
I20240502 14:07:51.849129 425 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:51.871789 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:51.883237 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:51.919322 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:51.944814 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:51.951403 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 14:07:51.974308 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:51.997936 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:52.037665 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:52.055100 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 14:07:52.070041 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:52.111405 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 14:07:52.111824 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:52.185148 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:52.270182 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:52.319187 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:07:52.339021 466 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:52.357229 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:52.382076 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:07:52.434738 425 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:52.438954 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:52.528501 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:07:52.535066 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:52.537917 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:52.544564 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:52.559598 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240502 14:07:52.576614 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:07:52.601220 443 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:07:52.601361 443 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:07:52.601791 443 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:07:52.602222 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 65 similar messages]
W20240502 14:07:52.602813 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:07:52.602972 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:07:52.603062 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:07:52.603214 443 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:07:52.901662 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:52.915894 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240502 14:07:52.916666 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:52.924450 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
I20240502 14:07:52.945865 466 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:52.979629 425 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:53.007685 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:53.168709 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 14:07:53.281306 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:53.363081 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
I20240502 14:07:53.405249 467 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:53.485862 487 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:53.539079 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 14:07:53.539710 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:53.541335 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:53.674022 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:53.681517 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:53.685242 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:53.694182 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:53.701911 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:53.809113 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:53.818373 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 14:07:53.873381 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:53.926803 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:07:53.929313 466 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:53.967505 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:54.004736 517 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:54.063639 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 14:07:54.145838 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:54.161926 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:54.232082 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:54.281709 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:54.304224 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:07:54.321151 466 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:54.325140 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 14:07:54.416536 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:54.430047 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 14:07:54.434417 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:07:54.434911 517 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:54.449375 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:54.471536 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 14:07:54.481913 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:54.485561 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:54.508046 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:54.512372 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 14:07:54.634406 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:54.731833 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 97 similar messages]
I20240502 14:07:54.747020 466 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:54.747534 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:54.764791 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:54.779423 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:54.815973 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:07:54.896651 517 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:54.923013 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:54.952337 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:54.958163 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:54.994040 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:55.034054 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:55.190750 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:55.203213 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:07:55.209959 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:55.264190 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240502 14:07:55.292946 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:07:55.325006 553 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:55.338490 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:55.362709 467 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:55.508560 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 14:07:55.686978 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:55.700047 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:55.741909 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:55.761750 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 14:07:55.811271 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:07:55.847733 570 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:55.922729 517 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:55.990881 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:56.002360 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 14:07:56.069013 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:56.184912 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:56.204048 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:56.281700 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:56.321756 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 14:07:56.340705 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 14:07:56.350735 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:56.376622 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:07:56.396903 570 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:56.423588 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:56.476564 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:07:56.476974 588 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:56.494181 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:56.502590 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:56.690264 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:56.703404 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:56.780613 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:56.810869 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:07:56.873543 570 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:56.874697 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:07:56.909109 588 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:56.935935 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 14:07:56.947245 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 14:07:56.951627 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:57.005160 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:57.058714 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:57.074033 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240502 14:07:57.085328 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:57.092804 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:57.145339 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:57.171690 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 14:07:57.203349 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:57.223841 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:57.266125 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:07:57.301697 587 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:57.308745 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:57.319114 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:57.452282 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:07:57.466393 517 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:57.471473 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:57.577754 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:57.631244 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:07:57.631783 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:57.668702 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:57.691529 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:07:57.701038 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:57.867770 629 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:57.899469 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
I20240502 14:07:57.910151 517 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:57.926438 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240502 14:07:57.977517 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:58.021219 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:58.129822 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 14:07:58.222601 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:58.239740 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 14:07:58.293108 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:07:58.374845 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:58.432346 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:58.457798 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240502 14:07:58.471801 629 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:58.533274 652 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:58.544773 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 14:07:58.550886 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:58.564671 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:07:58.611367 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:58.682431 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 14:07:58.696830 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:58.725991 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:07:58.742640 630 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:07:58.742787 630 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:07:58.743185 630 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:07:58.744165 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:07:58.744259 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:07:58.744323 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:07:58.744458 630 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:07:58.850617 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 14:07:58.867514 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:58.950579 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:58.958596 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:58.990445 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:07:59.021432 629 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:07:59.074210 671 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:59.087276 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:59.121690 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:59.127121 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:07:59.189239 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:07:59.245536 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:07:59.282094 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:07:59.322784 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240502 14:07:59.481199 629 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:59.512023 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 14:07:59.523625 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:59.526894 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:07:59.529028 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:07:59.529963 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:07:59.576617 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:07:59.638499 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 14:07:59.642577 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:07:59.678645 671 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:07:59.686489 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:07:59.714975 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:07:59.737474 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 14:07:59.738142 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:59.749599 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:07:59.783800 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 14:07:59.859457 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:07:59.919170 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:07:59.969082 665 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:00.033025 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:08:00.033077 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:08:00.163542 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:08:00.179592 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:08:00.248857 683 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:00.258236 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:08:00.366626 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:08:00.403188 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:08:00.438592 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240502 14:08:00.444927 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 14:08:00.480924 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:00.490484 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:00.553659 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
I20240502 14:08:00.572261 711 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:00.627496 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:08:00.707680 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:08:00.809204 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:08:00.862752 683 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:00.981053 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 14:08:01.033651 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:08:01.050590 665 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:01.051625 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:01.057548 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:08:01.083370 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:01.160218 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:01.275907 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 14:08:01.320566 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:08:01.445785 683 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:01.454103 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:01.462803 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 14:08:01.464941 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:01.482436 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:01.484400 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:08:01.496037 711 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:01.505995 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:08:01.566171 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:08:01.648330 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:01.666640 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:08:01.751925 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 14:08:01.752593 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 14:08:01.759833 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:08:01.859985 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:01.878998 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:01.957501 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:08:02.043327 747 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:02.046629 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:08:02.069869 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:02.073076 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:08:02.094947 754 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:02.148370 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:02.162344 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:02.174347 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 14:08:02.282655 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 14:08:02.339792 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:02.423053 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:02.453084 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:02.462463 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:08:02.470729 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:08:02.487190 747 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:02.552358 711 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:02.573211 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:02.641881 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 14:08:02.642627 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:02.647722 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 14:08:02.742422 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:08:02.776235 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 14:08:02.822436 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:08:02.880508 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:02.898952 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:08:02.921089 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240502 14:08:02.928988 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240502 14:08:02.974645 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:08:03.055826 747 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:03.082267 711 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:03.093317 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:03.096156 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:03.250541 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:03.402405 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:03.465137 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:08:03.466095 747 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:03.562556 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:03.571583 15506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 14:08:03.572186 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:03.636152 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 14:08:03.689409 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:08:03.694602 783 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:03.733153 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:03.843690 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:03.854797 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:08:03.927595 683 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:03.928634 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 14:08:04.047771 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:04.069370 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:04.082535 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:04.092070 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 14:08:04.167337 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:04.190258 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 14:08:04.209225 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:08:04.213335 810 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:04.213369 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:04.221387 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:04.248615 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:04.268774 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:04.273129 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:04.347468 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:04.375010 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:08:04.388068 683 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:04.396080 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 14:08:04.413638 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:04.512394 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:04.549772 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:04.712941 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 14:08:04.743927 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 98 similar messages]
I20240502 14:08:04.768328 747 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:04.822005 828 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:04.870395 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:04.917163 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:04.920562 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:04.927394 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:04.933342 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:04.940784 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:04.978348 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:05.044504 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 14:08:05.123338 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240502 14:08:05.147168 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:08:05.238679 747 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:05.262967 810 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:05.282490 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 14:08:05.291669 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:05.304721 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 14:08:05.311085 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:05.341243 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:08:05.353055 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:05.390925 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:05.454650 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:05.476276 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:08:05.684495 810 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:05.793916 683 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:05.868355 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:05.868765 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:05.883195 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:06.016669 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:06.034338 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:06.103927 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:06.142753 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
I20240502 14:08:06.188689 747 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:06.211037 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:06.222851 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
I20240502 14:08:06.239241 866 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:06.306851 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:06.309443 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:06.310648 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:06.460263 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:06.539611 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:06.614372 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 14:08:06.628988 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
I20240502 14:08:06.642920 866 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:06.651136 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:06.652143 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:06.667766 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:06.699601 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:06.705876 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:06.736150 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:08:06.805433 747 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:06.822824 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:06.830129 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240502 14:08:06.969549 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:06.992772 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:07.034725 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:07.035707 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240502 14:08:07.055126 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:07.059165 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:07.073724 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:08:07.110448 866 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:07.187206 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:07.209822 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 14:08:07.271203 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
I20240502 14:08:07.296315 747 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:07.422912 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:07.426349 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:07.471052 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:07.495769 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:08:07.552832 810 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:07.580767 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:07.602718 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:07.681278 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:08:07.707928 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:07.752775 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:07.756263 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:07.810573 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
I20240502 14:08:07.881036 910 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:07.981935 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:08.002722 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240502 14:08:08.011144 15637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240502 14:08:08.011735 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:08.070833 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:08:08.072273 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:08:08.176348 866 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:08.413053 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:08:08.427722 910 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:08.461025 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:08.464299 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:08.537978 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 14:08:08.549314 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:08.551936 810 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:08.566962 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:08.637267 15506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240502 14:08:08.732215 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:08.740159 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 14:08:08.787293 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:08.792660 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:08:08.821070 927 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:08.832659 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:08.889360 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:09.049276 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:09.058328 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:09.061677 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:09.070999 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240502 14:08:09.084251 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:09.143277 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:09.176424 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:08:09.176556 945 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:09.190730 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:09.214051 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 14:08:09.269793 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:08:09.295442 927 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:09.299625 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:09.433637 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:09.497308 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 14:08:09.521240 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 14:08:09.547037 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:09.581997 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:09.613570 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240502 14:08:09.668469 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:09.698433 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:09.722855 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:09.732059 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:08:09.740831 962 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:09.782521 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240502 14:08:09.798923 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 101 similar messages]
I20240502 14:08:09.807148 964 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:09.916079 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:09.946455 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:10.098943 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 14:08:10.102767 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:10.115744 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:10.156440 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:08:10.230522 927 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:10.231192 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:10.255610 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:08:10.311724 962 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:10.377470 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:10.440428 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:10.474184 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:10.497722 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:10.605650 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:10.634635 964 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:10.653679 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:08:10.656180 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 14:08:10.837705 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
I20240502 14:08:10.843418 962 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:10.873672 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:11.041965 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:11.116276 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:11.154999 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:08:11.173753 964 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:11.195935 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:11.268260 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 14:08:11.282905 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:11.283762 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:11.305949 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:11.322700 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:11.380666 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:11.387545 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:08:11.453100 1009 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:11.487035 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:11.570927 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:11.640436 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 14:08:11.641475 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:11.643029 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:11.646906 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:08:11.664099 964 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:11.682555 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:11.697829 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:08:11.721266 630 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:08:11.721380 630 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:08:11.721724 630 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:08:11.722622 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:08:11.722796 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:08:11.722880 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:08:11.723021 630 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:08:11.761471 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:11.799494 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:11.806805 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 14:08:11.822270 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:11.824368 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
I20240502 14:08:11.839867 1009 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:11.843936 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:11.864383 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240502 14:08:11.957759 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:08:12.039975 964 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:12.044119 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:12.151283 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:12.243809 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 14:08:12.246793 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:12.272729 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:12.311498 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:08:12.381299 1038 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:12.535302 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:08:12.540462 964 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:12.559322 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 14:08:12.591540 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:12.647891 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:12.683243 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240502 14:08:12.708843 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:12.775681 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 14:08:12.777032 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:08:12.907925 1038 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:12.920521 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:12.921662 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:12.955586 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:13.006371 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 14:08:13.058521 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:08:13.065032 15637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 154 similar messages]
W20240502 14:08:13.074445 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
I20240502 14:08:13.078898 927 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:13.117390 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:13.276839 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:08:13.320029 1051 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:13.480187 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:13.571169 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:13.592756 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:13.605701 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:13.606449 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:13.642516 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 14:08:13.644104 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:08:13.660817 927 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:13.672981 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:13.693336 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 14:08:13.706908 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:13.841804 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:08:13.849884 1082 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:13.942844 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:13.969740 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:14.035260 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:14.074229 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 14:08:14.086232 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:14.101938 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:14.141608 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:14.167197 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:14.183034 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:14.195113 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 14:08:14.199782 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:14.207473 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
I20240502 14:08:14.249555 964 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:14.260672 1082 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:14.295256 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:14.359778 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 14:08:14.407356 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:14.422672 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:14.467922 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:14.499469 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:14.555064 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:14.594529 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:14.686736 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
I20240502 14:08:14.731740 927 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:14.750852 1082 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:14.761911 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:14.835763 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:14.844244 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:08:14.943655 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:14.982451 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:15.004628 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:15.020334 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:15.039528 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 14:08:15.058255 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:15.082441 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240502 14:08:15.143195 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:08:15.147495 1051 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:15.262355 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:15.319808 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:08:15.338968 1122 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:15.410807 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:15.427682 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:15.438179 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 14:08:15.457218 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:15.607823 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
I20240502 14:08:15.623543 1082 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:15.838836 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:08:15.859925 15413 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a7981c3f46d8483caae82fbf2296e762 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 8484181c4ece483c8b408817fab1a917 (127.8.86.130:42261) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } }.
Pending config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "8484181c4ece483c8b408817fab1a917" member_type: VOTER last_known_addr { host: "127.8.86.130" port: 42261 } } peers { permanent_uuid: "8fea283177394bd28d7eacf4fb744d59" member_type: VOTER last_known_addr { host: "127.8.86.135" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "82d7b7201cfc431d90baca1ad0a397cf" member_type: NON_VOTER last_known_addr { host: "127.8.86.131" port: 41531 } attrs { promote: true } }
I20240502 14:08:15.919890 927 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:15.920898 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:15.922045 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:16.038668 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:16.050590 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:16.052242 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:16.059808 1082 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:16.185194 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:16.346506 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:08:16.359643 1122 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:16.385035 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:16.428402 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:16.448953 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:16.517729 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
I20240502 14:08:16.531983 1051 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:16.590670 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:16.620342 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:16.625607 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:16.628818 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 14:08:16.645231 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:16.650029 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:16.666584 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:16.677786 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 14:08:16.684279 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 14:08:16.724123 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:16.787717 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:16.799309 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:08:16.812664 927 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:16.870990 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 14:08:16.917564 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:16.930171 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:08:16.972208 1051 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:16.985188 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:16.992477 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:17.076208 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 14:08:17.124869 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:17.238853 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:17.242025 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:17.276894 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:17.288455 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:08:17.401413 927 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:17.405112 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:08:17.457746 1082 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:17.474968 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:17.526803 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:17.591991 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:17.599319 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:17.614509 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:17.648007 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:17.708714 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 14:08:17.751291 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240502 14:08:17.776804 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:17.791297 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:08:17.824895 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 14:08:17.896020 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:08:17.922555 927 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:18.013578 1082 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:18.071002 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 148 similar messages]
W20240502 14:08:18.143553 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 14:08:18.152602 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:18.286335 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:08:18.296258 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:18.309551 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:18.366727 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:08:18.506778 1193 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:18.567162 1214 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:18.608098 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:18.636163 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:18.645318 15505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240502 14:08:18.677346 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:18.739338 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:18.764755 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:18.879317 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 14:08:18.930858 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 14:08:18.942451 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:08:18.947479 1082 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:18.996034 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 14:08:19.054790 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:19.068420 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:19.074247 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:08:19.102958 1225 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:19.162171 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:19.203015 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:19.216001 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:19.219192 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:19.229624 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 14:08:19.283845 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:19.334007 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
I20240502 14:08:19.349351 1082 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:19.483948 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:19.496539 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:08:19.509999 1225 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:19.511056 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:19.531548 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:19.582756 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:19.619844 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:19.625998 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 14:08:19.632874 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:19.751399 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:19.781273 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:19.807473 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:19.817466 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:19.817507 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:19.901924 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 103 similar messages]
I20240502 14:08:19.928731 1082 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:20.057025 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:20.073397 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:20.105700 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:08:20.129186 1252 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:20.192536 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:20.219652 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 14:08:20.244269 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:20.323443 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:20.327024 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 14:08:20.398869 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:08:20.431905 1082 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:20.433619 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 14:08:20.490921 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240502 14:08:20.528519 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:08:20.653005 1271 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:20.709757 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:08:20.758687 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:20.818850 1265 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:20.851136 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:20.884249 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:21.008822 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:21.090265 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:21.137507 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:21.194092 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 14:08:21.213990 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 14:08:21.228483 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:21.253698 1271 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:21.260344 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:21.296344 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:21.309321 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
I20240502 14:08:21.329562 1265 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:21.333860 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:21.353390 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:21.563297 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:21.572801 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:21.604756 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:21.609860 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:21.690490 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
I20240502 14:08:21.753985 1082 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:21.773530 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:21.833338 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:08:21.846752 1304 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:21.854060 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:21.884490 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:21.888399 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:21.912902 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 14:08:21.966147 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:21.983687 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:21.994118 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 14:08:22.023131 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:22.119438 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:08:22.163115 1265 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:22.192720 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:22.194278 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:22.343050 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:22.381965 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:22.388484 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:22.414937 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:22.448786 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:08:22.457504 1321 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:22.555390 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:22.680699 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:22.684329 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:08:22.706239 1082 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:22.727488 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240502 14:08:22.770157 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:22.772213 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 14:08:22.798002 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:22.830946 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:08:22.851351 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
I20240502 14:08:22.852989 1271 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:22.942509 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 14:08:23.017253 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:23.079263 15637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240502 14:08:23.109791 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:08:23.184844 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:23.215636 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:23.235863 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:08:23.291596 1344 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:23.377445 1321 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:23.578224 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:23.617033 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:23.620029 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:23.621683 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:23.627876 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:23.662585 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 14:08:23.678099 15506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 141 similar messages]
I20240502 14:08:23.695633 1344 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:23.710498 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:23.732939 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 14:08:23.792481 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
I20240502 14:08:23.809839 1271 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:23.847204 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:23.857931 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:23.912052 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:23.955722 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 14:08:23.961887 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:23.987524 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:24.091347 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:24.109400 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:24.210867 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:24.220580 1271 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:24.248260 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:24.265136 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:24.267735 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 14:08:24.279361 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:08:24.281075 1344 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:24.407305 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:24.455089 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:24.503340 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:24.513217 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:08:24.676949 1344 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:24.688974 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 14:08:24.713146 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:24.713192 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:24.761646 1381 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:24.792037 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:24.822256 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:24.884038 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:24.908965 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:08:24.909708 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:24.974251 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:25.013422 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:25.016979 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:25.032830 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 14:08:25.055959 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
I20240502 14:08:25.085196 1344 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:25.088409 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:25.093353 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:25.157980 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:08:25.218377 1381 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:25.314927 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 14:08:25.433867 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:08:25.468730 1344 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:25.548846 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:25.579118 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240502 14:08:25.600611 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:08:25.668188 1271 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:25.674425 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:25.773320 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:25.775241 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:08:25.886921 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:08:25.899592 1344 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:25.971539 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 14:08:26.004166 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:26.028893 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:26.069144 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:26.109467 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:26.109915 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:08:26.113535 1381 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:26.201911 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:26.295389 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:26.377959 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
I20240502 14:08:26.436161 1082 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:26.501279 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:26.514214 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 14:08:26.533936 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:26.548565 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:26.551429 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 14:08:26.567952 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:26.632014 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:08:26.646080 1271 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:26.662688 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:26.678936 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:26.744760 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 14:08:26.768018 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:26.821578 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:08:26.932360 1421 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:26.969483 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:26.972687 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:26.983350 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:26.993602 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:27.008572 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:08:27.066152 1271 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:27.074224 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:27.182140 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 14:08:27.262981 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:08:27.329030 1421 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:27.346024 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:27.350468 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:27.406445 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:08:27.454344 1271 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:27.479084 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:27.526392 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:27.566403 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:27.572206 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:27.667228 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 14:08:27.676486 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:27.686192 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
I20240502 14:08:27.706487 1421 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:27.708138 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:27.746055 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:27.766506 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 14:08:27.815443 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:27.930997 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:27.940198 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 14:08:28.001525 1271 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:28.058332 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:28.123235 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240502 14:08:28.176826 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:08:28.216374 1474 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:28.217347 15636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240502 14:08:28.372200 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:28.408469 1282 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:08:28.408620 1282 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:08:28.409188 1282 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:08:28.410120 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:08:28.410223 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:08:28.410346 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:08:28.410478 1282 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:08:28.415611 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:08:28.415851 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
I20240502 14:08:28.419601 1381 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:28.540822 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:28.620327 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:28.633297 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:28.695374 15505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 14:08:28.731889 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:08:28.767203 1492 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:28.771330 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:28.840855 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:28.842790 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:28.846537 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:08:28.846894 1381 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:28.866173 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:28.875299 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 14:08:28.902112 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:28.992761 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:29.016034 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:29.060359 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240502 14:08:29.122960 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:29.177803 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
I20240502 14:08:29.224905 1492 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:29.248793 1271 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:29.290814 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:29.306600 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:29.320125 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 14:08:29.371574 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:29.383716 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:29.452937 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:29.502866 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:29.517560 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:29.526959 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:29.632968 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:08:29.686542 1271 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:29.792198 1492 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:29.853003 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:29.876281 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:29.877923 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:29.907816 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:29.928575 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 14:08:29.934003 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 14:08:29.940960 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240502 14:08:29.956450 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:30.047559 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:30.142632 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:30.156781 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 14:08:30.181588 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 14:08:30.187705 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:30.237571 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:08:30.250267 1492 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:30.288332 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:30.299731 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:30.306133 1531 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:30.306790 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:30.311414 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:30.323699 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:30.448277 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
I20240502 14:08:30.726663 1531 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:30.778081 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:08:30.810437 1520 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:30.826260 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:30.876744 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 14:08:30.885591 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:30.956629 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:08:31.061524 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:31.134218 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
I20240502 14:08:31.181448 1531 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:31.203410 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:31.206831 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:08:31.209988 1520 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:31.281865 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:31.377390 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:31.382124 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:31.404515 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:31.470047 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:31.488999 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:31.523300 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:31.564977 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 14:08:31.583636 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
I20240502 14:08:31.649031 1492 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:31.742759 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:08:31.747864 1531 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:31.772075 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:31.804252 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:31.827728 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:31.894349 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 14:08:31.927116 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:31.976846 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:31.978003 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:31.978963 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:31.995355 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:32.006108 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:32.115108 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:08:32.200804 1492 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:32.211000 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:32.281555 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:08:32.358564 1531 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:32.400803 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:32.413370 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:32.435838 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:32.469064 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 14:08:32.502727 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 14:08:32.571200 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 14:08:32.645076 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:32.645836 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:32.681666 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:32.714577 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 14:08:32.714951 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:32.715437 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:32.732822 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:08:32.761924 1589 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:32.825475 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:32.850615 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:32.887302 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:32.931427 1590 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:32.978964 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 14:08:33.150465 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240502 14:08:33.232831 15637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 148 similar messages]
I20240502 14:08:33.259989 1589 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:33.305682 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:33.390650 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:33.402523 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:08:33.455644 1590 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:33.482268 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:33.484436 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 14:08:33.572130 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:33.664880 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 14:08:33.698920 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240502 14:08:33.712317 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:08:33.764277 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:33.808214 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:08:33.849831 1608 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:33.864059 1637 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:33.927722 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:33.953555 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 14:08:34.051518 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:34.053894 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:34.062592 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:34.067411 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 14:08:34.094681 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:34.104709 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:34.153316 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:34.193908 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:34.253168 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 14:08:34.273871 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:34.308537 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:08:34.335704 1590 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:34.458595 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:08:34.485857 1653 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:34.507663 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:34.544150 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:34.584487 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:34.639065 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:34.658022 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:34.752996 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:34.772521 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 14:08:34.779815 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:34.822530 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:34.836745 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:08:34.958547 1664 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:34.986949 1653 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:35.018792 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 14:08:35.068631 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240502 14:08:35.071267 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:35.144078 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:35.161592 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:35.177783 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:35.177811 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:35.179083 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:35.212030 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 14:08:35.221711 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 14:08:35.260701 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:35.327762 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:35.379642 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:08:35.380473 1637 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:35.404940 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:08:35.552006 1664 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:35.694438 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:35.700074 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:35.824553 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240502 14:08:35.852720 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:35.858986 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 14:08:35.909346 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:08:35.945528 1653 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:35.964407 1590 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:36.027261 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:08:36.056288 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:36.134948 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:36.171918 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:36.347594 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 14:08:36.350319 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
I20240502 14:08:36.452531 1664 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:36.455022 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:36.472858 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:08:36.493306 1705 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:36.494640 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 14:08:36.565455 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:36.568367 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:36.665236 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 14:08:36.673833 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:36.698179 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:36.720006 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:36.728195 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:36.780128 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:36.808861 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:36.883031 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:08:36.921779 1590 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:36.982096 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:36.991501 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:08:36.992336 1653 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:37.106500 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:37.131907 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 14:08:37.148898 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:37.212025 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:37.235481 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:37.236701 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:37.308921 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:37.320082 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:37.351943 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:37.431818 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:08:37.482012 1705 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:37.483183 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 14:08:37.511366 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:08:37.529848 1664 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:37.604938 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:37.605443 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:37.674682 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:37.689728 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:37.733435 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 14:08:37.747165 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:37.754086 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:37.772524 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:37.869755 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 14:08:37.883535 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:08:37.970247 1664 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:38.023136 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 14:08:38.073153 1752 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:38.101356 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240502 14:08:38.111440 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:38.138883 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:38.229468 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 14:08:38.266005 15637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 146 similar messages]
W20240502 14:08:38.313918 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:08:38.328397 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:08:38.380846 1590 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:38.491925 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:38.523170 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:38.539513 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:38.656632 1769 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:38.686020 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240502 14:08:38.701396 15505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 14:08:38.732215 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
I20240502 14:08:38.762506 1590 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:38.937295 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:38.973250 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 14:08:38.984334 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:39.000519 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:39.014907 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:39.024420 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:08:39.054133 1769 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:39.065927 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:39.112179 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:39.167054 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:39.219249 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:39.282457 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 14:08:39.303998 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:39.320322 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:08:39.371045 1664 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:39.398727 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:39.411809 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:39.435161 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 14:08:39.459568 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:39.539469 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:39.557935 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:08:39.583202 1769 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:39.614029 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:39.687340 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:39.714776 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:39.734434 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:39.748965 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:39.771171 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 14:08:39.855482 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:39.887660 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:08:39.891695 1664 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:40.008142 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:40.051481 1752 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:40.070361 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240502 14:08:40.071040 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:40.092828 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:40.093279 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 14:08:40.162726 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:40.169278 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:40.256690 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 14:08:40.284843 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:40.292410 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:40.326398 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:40.421340 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:40.436878 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:40.475834 1664 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:40.498730 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 14:08:40.523766 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
I20240502 14:08:40.564335 1752 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:40.613843 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:40.631860 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:40.899818 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:40.900957 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:08:40.950598 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:40.961102 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:41.044765 1664 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:41.171343 1752 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:41.171988 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 14:08:41.241372 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 14:08:41.313097 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:41.319705 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 14:08:41.324110 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:41.360427 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:41.431963 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:41.437238 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:41.565563 1664 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:41.590137 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:41.628736 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:08:41.664446 1752 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:41.761392 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:41.763062 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 14:08:41.775803 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:41.825034 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:41.856387 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:41.857759 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:41.910645 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:41.947384 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:42.001226 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:42.023536 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:42.039116 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
I20240502 14:08:42.079123 1752 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:42.103875 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:42.107316 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:42.135854 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:08:42.150367 1664 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:42.187067 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:42.274338 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 14:08:42.303701 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:42.305548 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:42.325145 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:42.390321 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:42.406466 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:42.448778 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:42.619720 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:08:42.628142 1867 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:42.700314 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:08:42.700762 1878 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:42.721783 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:42.735092 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 14:08:42.794122 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:42.863997 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:42.893774 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:42.905275 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:42.917182 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 14:08:43.013976 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
I20240502 14:08:43.042313 1664 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:43.042382 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 14:08:43.102218 1878 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:43.130692 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240502 14:08:43.249774 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:43.296329 15635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240502 14:08:43.454592 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:43.474449 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:43.490130 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:08:43.509395 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:43.543529 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 14:08:43.543915 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 14:08:43.567160 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:43.602488 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:08:43.646126 1664 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:43.689054 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
I20240502 14:08:43.703816 1878 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:43.763203 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240502 14:08:43.832442 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:08:43.888981 1856 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:08:43.889147 1856 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:08:43.889726 1856 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:08:43.890900 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:08:43.890985 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:08:43.891063 15897 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:08:43.891188 1856 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:08:43.927381 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:43.947103 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:43.972271 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:43.987789 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:43.989025 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:44.031375 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:44.125942 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:08:44.184463 1926 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:44.202590 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:08:44.307993 1908 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:44.332120 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:44.384236 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 14:08:44.449012 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:44.472980 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:44.476462 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:44.485729 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:44.538601 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:44.572469 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:44.606765 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:44.606956 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:08:44.662667 1926 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:44.701184 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:44.750761 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:08:44.757674 1908 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:44.766045 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 14:08:44.843787 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:44.887965 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 14:08:44.889974 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:44.909087 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:44.910043 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:45.049842 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:45.055503 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:45.063593 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:08:45.078779 1664 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:45.155192 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240502 14:08:45.183915 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:45.234340 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
I20240502 14:08:45.256675 1878 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:45.303380 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 14:08:45.389582 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:45.446596 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:45.452790 15413 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 14:08:45.453483 15413 catalog_manager.cc:4573] TS 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b0cd21dc897b41aca9118be6df48cc0c: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111)
W20240502 14:08:45.487862 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
I20240502 14:08:45.516412 1926 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:45.598228 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:08:45.617839 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:45.624732 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:45.657258 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 14:08:45.680423 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240502 14:08:45.736740 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:08:45.804637 1878 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:46.018347 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:46.045982 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:08:46.077817 1980 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:46.097468 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:46.135715 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240502 14:08:46.171579 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:46.188213 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:08:46.275524 1878 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:46.290802 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:46.308357 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 14:08:46.344570 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:46.351691 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:46.476992 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2886dfbec1e141198b0cf7a435fac498: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:08:46.525519 1926 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:46.589897 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:46.625900 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:46.637782 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:46.651464 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:46.688081 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:08:46.689321 1908 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:46.812826 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:46.814266 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:46.836061 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:46.854899 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:46.895488 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:08:46.914783 1926 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:46.974141 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:46.976261 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 14:08:47.053395 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:47.079416 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:47.119053 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:47.129961 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:47.213819 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:08:47.278122 2012 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:47.331884 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:47.338441 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:47.356644 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:47.366814 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:47.380151 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:47.394534 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:47.401705 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 14:08:47.443709 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:47.444912 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:08:47.510871 1980 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:47.534871 1856 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:08:47.535032 1856 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:08:47.535509 1856 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:08:47.536538 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
W20240502 14:08:47.536638 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
I20240502 14:08:47.536726 15898 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:08:47.536870 1856 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:08:47.580066 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 14:08:47.595085 15414 catalog_manager.cc:4573] TS f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 48fbc670ec504d26a11bf1b9fbd78981: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:08:47.661844 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:08:47.709399 1908 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:47.785871 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:47.813871 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 14:08:47.819607 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:47.832038 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 14:08:47.893574 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:47.904666 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:47.978364 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:48.042219 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
I20240502 14:08:48.043509 1926 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:48.060094 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 14:08:48.105841 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:48.115460 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240502 14:08:48.136552 1908 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:48.195483 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 14:08:48.334146 15637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 152 similar messages]
W20240502 14:08:48.373716 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:08:48.471491 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:48.486892 1926 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:48.523414 1908 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:48.568266 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:48.664829 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:48.750008 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:48.790128 15506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240502 14:08:48.792666 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 14:08:48.862454 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:48.911727 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:08:48.982877 1980 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:49.004338 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 14:08:49.009097 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:49.069891 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:49.071084 2059 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:49.071489 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:49.085271 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:49.241765 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:49.277701 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:49.298648 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:49.328193 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:49.336441 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:49.424085 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:49.449409 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 14:08:49.457522 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:08:49.462714 1908 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:49.584420 1980 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:49.621006 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:49.642647 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:49.647354 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:49.699579 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:49.719991 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:49.747184 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:49.792172 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:49.839350 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:49.926648 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 14:08:49.948211 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:49.952409 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:49.964275 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:50.002259 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:08:50.046830 1908 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:50.082365 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:50.127174 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
I20240502 14:08:50.131217 1926 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:50.157537 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111) [suppressed 97 similar messages]
W20240502 14:08:50.190738 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:50.207433 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:50.254704 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 14:08:50.268205 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:50.286988 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:50.340950 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:50.478003 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
I20240502 14:08:50.533866 1908 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:50.578061 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240502 14:08:50.581421 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
I20240502 14:08:50.583338 1926 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:50.605989 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 14:08:50.611218 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:50.616451 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:50.685166 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:08:50.708846 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:50.917757 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:08:50.955916 2085 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:51.071406 1980 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:51.160956 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:51.181754 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:51.313083 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
I20240502 14:08:51.396265 2085 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:51.436756 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:51.443571 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 14:08:51.461818 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:51.477078 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:51.520751 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:51.543588 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:51.616317 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:08:51.648340 2130 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:51.699347 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:51.725193 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:51.751451 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:51.781621 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:51.854936 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:51.871601 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:51.873776 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:51.934224 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 14:08:51.942621 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:08:51.985895 2085 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:51.998500 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:52.045547 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:52.066025 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:52.118885 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:08:52.147979 1980 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:52.224795 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:52.261302 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:52.298533 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:52.317993 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 14:08:52.323153 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:52.333544 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:08:52.453322 2085 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:52.495971 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:52.572767 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:52.598114 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:52.613286 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:52.675268 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:08:52.684041 2130 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:52.688535 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 14:08:52.732383 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:52.748389 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:52.756857 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 14:08:52.768273 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:52.807322 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:52.929369 2085 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:52.981395 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:53.072273 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240502 14:08:53.076324 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:08:53.095351 2130 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:53.105746 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:08:53.109835 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 14:08:53.134339 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 14:08:53.179677 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 14:08:53.202935 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 14:08:53.253381 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:53.305552 2136 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:53.354367 15637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240502 14:08:53.597548 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:53.629501 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:08:53.651067 2180 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:53.746940 2136 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:53.812414 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:53.816365 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240502 14:08:53.905748 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:53.924269 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 14:08:53.945179 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 14:08:53.947187 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:54.027241 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:54.044420 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:08:54.143848 2130 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:54.147511 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:54.155169 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:54.157822 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:54.197353 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 14:08:54.223673 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:54.267128 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:54.299448 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:54.362864 2202 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:54.394886 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:54.457664 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:54.460412 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:54.462832 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:54.465873 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:08:54.598875 2130 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:54.621418 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:54.628113 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:54.630077 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:54.679294 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:54.748234 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:54.753952 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:54.835907 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:08:54.836879 2136 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:54.856678 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:54.907192 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 14:08:54.958799 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:54.985823 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:55.009382 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:08:55.101004 1856 raft_consensus.cc:482] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:08:55.101157 1856 raft_consensus.cc:504] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4d0f8f311e834d81bd54b9f8e7923bcd" member_type: VOTER last_known_addr { host: "127.8.86.132" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "3c3658c946994e2ba9f41b64aa327d25" member_type: VOTER last_known_addr { host: "127.8.86.134" port: 34379 } attrs { promote: false } } peers { permanent_uuid: "f688370715454d6e814a78fcf85ac5c3" member_type: VOTER last_known_addr { host: "127.8.86.133" port: 42151 } attrs { promote: false } } peers { permanent_uuid: "2254b3ee3db647598d781e467584f78d" member_type: NON_VOTER last_known_addr { host: "127.8.86.136" port: 38631 } attrs { promote: true } }
I20240502 14:08:55.101567 1856 leader_election.cc:288] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379), f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151)
W20240502 14:08:55.102569 15898 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111)
W20240502 14:08:55.102681 15897 leader_election.cc:334] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111)
I20240502 14:08:55.102754 15897 leader_election.cc:302] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d0f8f311e834d81bd54b9f8e7923bcd; no voters: 3c3658c946994e2ba9f41b64aa327d25, f688370715454d6e814a78fcf85ac5c3
I20240502 14:08:55.102877 1856 raft_consensus.cc:2726] T 387bdd8ae3b34f588e2170b061a5bc05 P 4d0f8f311e834d81bd54b9f8e7923bcd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:08:55.108389 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
I20240502 14:08:55.115262 2221 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:55.155328 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:55.158221 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240502 14:08:55.197999 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:08:55.221244 2202 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:55.265843 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:55.355593 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:55.355736 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240502 14:08:55.362274 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:55.395079 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 14:08:55.425815 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:55.501742 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:08:55.578884 2221 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:55.617700 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 14:08:55.656899 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
I20240502 14:08:55.663123 2202 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:55.676633 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 14:08:55.734474 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:55.793540 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:08:55.809285 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:56.025168 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:56.080072 2202 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:56.169926 2252 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:56.219758 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:56.242841 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:56.243198 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:56.245455 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:56.323359 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 14:08:56.344836 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:08:56.474084 2202 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:56.552886 2252 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:56.554683 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:56.561082 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:56.577831 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 14:08:56.605757 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:56.609925 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 14:08:56.637316 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:56.707621 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:56.788658 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:56.826074 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:08:56.830221 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:08:56.872789 2202 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:56.887290 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:56.893395 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:56.920418 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:56.965903 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:08:57.085264 2276 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:57.106201 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:57.110072 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:57.128846 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:57.134825 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:57.212033 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:08:57.256819 2202 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:57.276435 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:57.316998 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:57.341451 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 14:08:57.389175 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:57.510886 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:57.557962 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
I20240502 14:08:57.586380 2276 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:57.651355 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:57.652352 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:57.680114 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:08:57.685405 2136 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:57.710983 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:57.742908 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:57.749696 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240502 14:08:57.800627 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 14:08:57.864308 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:57.984930 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 14:08:57.996963 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:08:58.023720 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 14:08:58.071825 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:08:58.083851 2276 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:58.244483 2309 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:58.266623 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:08:58.297876 15899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 14:08:58.298523 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:58.309022 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 14:08:58.359400 15637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240502 14:08:58.390012 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:08:58.563072 2252 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:58.621850 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:08:58.678249 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:58.710036 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:58.722713 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:08:58.727715 2136 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:58.769513 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:08:58.774231 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:08:58.783257 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 14:08:58.787866 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 14:08:58.824880 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:08:58.826261 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:08:58.834903 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240502 14:08:58.870898 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:08:59.019659 2252 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:59.085508 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:08:59.103196 2136 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:59.146495 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 14:08:59.151793 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:59.181320 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:59.192168 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:59.238866 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:08:59.429569 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:08:59.442176 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:59.476655 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:08:59.499749 2276 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:08:59.580788 2136 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:08:59.608569 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:08:59.617287 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:08:59.625196 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:08:59.645820 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:59.673699 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:08:59.681344 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:08:59.687995 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:08:59.710882 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:08:59.729265 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:08:59.847890 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 14:08:59.948348 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 14:08:59.967149 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:09:00.009172 2354 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:09:00.046892 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240502 14:09:00.085417 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:09:00.105281 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:09:00.162714 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240502 14:09:00.167804 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:09:00.182683 2360 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:09:00.224721 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 14:09:00.235697 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:09:00.246734 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:09:00.251533 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:09:00.307948 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
I20240502 14:09:00.397818 2354 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:09:00.410498 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:09:00.418035 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 14:09:00.561220 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:09:00.631338 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:09:00.665092 2360 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:09:00.697054 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:09:00.721814 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:09:00.771481 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:09:00.777899 2354 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:09:00.829033 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 14:09:01.074194 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:09:01.148725 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:09:01.162278 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:09:01.163292 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 14:09:01.164999 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:09:01.165586 2136 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:09:01.173583 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:09:01.201593 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:09:01.219367 2276 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:09:01.236734 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:09:01.263294 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:09:01.292654 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:09:01.474000 15767 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 14:09:01.581863 15768 consensus_peers.cc:485] T 28e371c1a999431499e09479e8f5b44b P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:09:01.604445 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 14:09:01.641252 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:09:01.693045 2354 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:09:01.712491 2136 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:09:01.734125 15899 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:09:01.818145 15506 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:09:01.833262 15897 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:09:01.932497 15505 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:09:01.936663 15504 consensus_peers.cc:485] T b0cd21dc897b41aca9118be6df48cc0c P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:09:02.006824 15635 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:09:02.077236 15504 consensus_peers.cc:485] T bb9f100600a246708457280d91826fce P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:09:02.120887 15636 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:09:02.130829 15766 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:09:02.141796 15768 consensus_peers.cc:485] T 2886dfbec1e141198b0cf7a435fac498 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:09:02.143512 15899 consensus_peers.cc:485] T e48dda3e62e14e46b03d79648e3e2773 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:09:02.180182 15635 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:09:02.204950 15636 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 14:09:02.245522 15636 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:09:02.246351 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:09:02.251022 2414 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 14:09:02.263976 2354 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:09:02.376031 15506 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:09:02.431797 15637 consensus_peers.cc:485] T 48fbc670ec504d26a11bf1b9fbd78981 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:09:02.501672 15768 consensus_peers.cc:485] T 22eef55211ca4f65979022d2f4ae829f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:09:02.596791 15505 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240502 14:09:02.612613 15635 consensus_peers.cc:485] T e51db70f5f284c4da3c1669fddb41b00 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 14:09:02.658614 15635 consensus_peers.cc:485] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:09:02.673993 15899 consensus_peers.cc:485] T 2d92e5885bf94c3e8e0e4dee40017699 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:09:02.707473 2354 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:09:02.722218 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:09:02.756258 2414 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:09:02.794934 15768 consensus_peers.cc:485] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:09:02.802518 15504 consensus_peers.cc:485] T d81f14b7c1d1468196d0fe6d67fe93a7 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:09:02.813089 15637 consensus_peers.cc:485] T 1f80c8590e994a10a9b7c6c174e28173 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:09:02.819339 15766 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240502 14:09:02.835057 15767 consensus_peers.cc:485] T 1c004727e7c64f33bc9492184b77fde9 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240502 14:09:02.884965 15766 consensus_peers.cc:485] T 02d3da05d82340178e27218c2ce56b67 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 14:09:03.016559 15899 consensus_peers.cc:485] T 1a5e33f337bd47b2a6b11b29a5eacad8 P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:09:03.036398 15504 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:09:03.183607 15637 consensus_peers.cc:485] T b07f4db6f82e4a56accdbf1bbbc5f965 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:09:03.197551 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
I20240502 14:09:03.209666 2354 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:09:03.342872 15897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 14:09:03.346670 15637 consensus_peers.cc:485] T 6e76bd0ecd3e4c168e141fb7984af3ec P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:09:03.353323 2414 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:09:03.381790 15637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111) [suppressed 151 similar messages]
W20240502 14:09:03.526862 15504 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240502 14:09:03.567742 15504 consensus_peers.cc:485] T 186a946321e445b58b0097c658847ebe P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:09:03.572402 15505 consensus_peers.cc:485] T 1f562ad00ec34a8d9184671ed55d84d1 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer f688370715454d6e814a78fcf85ac5c3 (127.8.86.133:42151): Couldn't send request to peer f688370715454d6e814a78fcf85ac5c3. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.133:42151: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
I20240502 14:09:03.641661 2442 raft_consensus.cc:1013] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 [term 1 LEADER]: attempt to promote peer 82d7b7201cfc431d90baca1ad0a397cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:09:03.689949 15637 consensus_peers.cc:462] T b6d365c10c76416a944ec8fe409e3dc5 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111)
W20240502 14:09:03.729946 15768 consensus_peers.cc:485] T 7c57cab68b764db0917ed98e74757ff0 P 82d7b7201cfc431d90baca1ad0a397cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:09:03.739066 15506 consensus_peers.cc:485] T 9870bc797f184a6284ff6ca4f78b3826 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:09:03.767989 15897 consensus_peers.cc:485] T 837e8b1235a44131bdaef24b763d1cac P 4d0f8f311e834d81bd54b9f8e7923bcd -> Peer 2254b3ee3db647598d781e467584f78d (127.8.86.136:38631): Couldn't send request to peer 2254b3ee3db647598d781e467584f78d. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.136:38631: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:09:03.807102 15637 consensus_peers.cc:485] T 436c127af87c49e8b7bf9b6571f7dc30 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:09:03.858006 2460 raft_consensus.cc:1013] T 03dd735113ab4ea6bdb696b11a4efd7f P 82d7b7201cfc431d90baca1ad0a397cf [term 1 LEADER]: attempt to promote peer 1eb0e3a6c782469fa8110f6bb6a585cf: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 14:09:03.878193 15504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240502 14:09:03.883127 15635 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:09:03.890538 15637 consensus_peers.cc:485] T a7981c3f46d8483caae82fbf2296e762 P 8484181c4ece483c8b408817fab1a917 -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn't send request to peer 8fea283177394bd28d7eacf4fb744d59. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.135:46245: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:09:03.926106 15504 consensus_peers.cc:485] T dfdcf1124e8a474facc14c45e8963568 P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 3c3658c946994e2ba9f41b64aa327d25 (127.8.86.134:34379): Couldn't send request to peer 3c3658c946994e2ba9f41b64aa327d25. Status: Network error: Client connection negotiation failed: client connection to 127.8.86.134:34379: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240502 14:09:03.943572 15506 consensus_peers.cc:485] T 619a8ec66ba74ecea689c17b4734253a P 1eb0e3a6c782469fa8110f6bb6a585cf -> Peer 8fea283177394bd28d7eacf4fb744d59 (127.8.86.135:46245): Couldn'